Comparative Analysis of Anatomical Items

ABSTRACT

A computer-implemented medical visualization method includes identifying a three-dimensional model of an anatomical item of a particular mammal; displaying a moving animation of the three-dimensional model, the moving animation created from multiple frames from imaging of the anatomical item over a short time period to capture movement of the anatomical item; displaying one or more non-moving views of the three-dimensional model while the moving animation is being displayed; and in response to receiving inputs from a user, changing the displayed moving animation and the one or more non-moving views automatically in coordination with each other.

RELATED APPLICATIONS

The present applications claims priority to U.S. Provisional Application61/865,596, titled “Computer Modeling of Anatomical Items,” and U.S.Provisional Application 61/865,407, titled “Multi-Touch Interface,”\both filed Aug. 13, 2013, the contents of which are incorporated byreference in their entirety.

TECHNICAL FIELD

This document relates to systems and techniques for viewing andmanipulating models of body parts, such as human and other animalorgans, so as to improve computer operation.

BACKGROUND

Medical imaging technologies allow physicians and researchers tovisualize internal items inside the body of an animal subject. Suchimaging technologies generally provide a two-dimensional representationof a slice through the body at a particular depth, or a projection of anentire body onto a two-dimensional plane (e.g., with x-rays). Typicalmodern, digital imaging techniques include magnetic resonance imaging(MRI) and computed tomography (CT) scans. In addition to producingimages that can be colored for more ready interpretation, such scans canalso produce data files that can be automatically analyzed andmanipulated in various manners. For example, imaging data may be used toconstruct 3D jigs for use in dental and orthopedic procedures for aparticular patient.

SUMMARY

This document describes systems and techniques by which users of acomputer-based visualization system can manipulate and visualizeanatomical items, such as organs (e.g., hearts and connectedvasculature) from humans and other animals. Models of such anatomicalitems may be obtained from medical imaging of an actual subject, such asby obtaining digital data generated by magnetic resonance imaging (MRI)machines and computer tomography (CT) machines. Such data may berepresented by a number of two-dimensional images at different layers ofthe item, as the imaging equipment scanned layer-by-layer through theitem. The systems described here can identify common locations in imagesor models in the anatomical item from the two different medical imagingmodalities, may generate a combination of the data, and may generate a3D mesh of lines or points that substantially reflect the dimensions ofthe anatomical item. The systems may then generate a 3D model from themesh.

In addition, imaging data may be captured at different points in themovement of the anatomical item, such as at a dozen or so points in timeduring a cycle of a heartbeat where the anatomical item is a heart.Those data from different times may be used to generate athree-dimensional model for each point in time, and thosethree-dimensional models may be joined together to create afour-dimensional model (i.e., three spatial dimensions plus time).

Such models may permit researchers or clinicians to analyze andmanipulate an anatomical item in convenient, relatively inexpensive, andintuitive manners. As a simple example, a physician can order imagingfor a particular patient, and may don stereoscopic goggles to view thepatient's heart from different angles. The physician may also introducea cutting plane to the model so as to see the patient's heart from theinside—e.g., cutting horizontally across the atria, and then lookingdown into the heart from the top so as to see the motion and sealing ofthe heart valves. In some implementations, multiple models of the sameanatomical item may be displayed to assist a user in betterunderstanding the item. For example, imaging may be performed on apatient both before and after an operation or other medical procedure isperformed on the patient, such as the implanting of a medical devicelike a stent or cardioverter. Three-dimensional or four-dimensionalmodels of the anatomical item may then be displayed in 3D space adjacentto each other, and action by a user (e.g., a treating physician) withrespect to the item may be reflected concurrently in both models. Forexample, if a user applies a cutting plane to see inside an atrium orinside certain vasculature in the post-op model, the pre-op model mayhave the plane applied in the same location (with each of the modelshaving been positioned and oriented relative to a common base point).Similarly, a 4D model may be displayed concurrently with one or more 3Dmodels that are based off the same imaging data (or where some of themodels may use imaging data from a different time period for the sameorgan or other body part). The 3D models may be considered non-moving inthat they are not animated, though all of the models may be panned,zoomed, rotated, or clipped in coordination with each other in responseto user inputs. In this manner, for example, a physician could see theoperation of a patient's heart valves in a 4D view and simultaneouslyview a 3D view that shows the heart with the valves closed—so that thephysician can see the degree to which the valves close and can view arepeating cycling (similar to display of a GIF file that cycles) to tryto determine mechanically why the valve is closing the way it is—withthe 3D presentation at the particular point in time displayed adjacentthe 4D representation that cycles over time.

Also, a physician or other user may select different points in space ona projected 3D model, and the system may compute the distance betweenthose points, may snap to the edges of an opening near the points andcompute the diameter, area, or radius of the opening, and/or may performother similar measurements and may communicate the measurements to theuser, such as by annotating the model with alpha-numeric characters thatexpress the measurement. The measurement may also be expressed for amoving 4D model, such as via numbers that represent a maximum andminimum dimension across the range of motion at the points where theuser made the selections.

Moreover, the interaction of the model with external items introduced tothe model may be reviewed and studied. For example, a medical device maybe introduced into the anatomical item, such as by inserting the deviceinto an open passage in the item. As the device is moved, a physician orresearcher can determine whether it interferes with tissue in the model,and to what extent it interferes. For example, a system may beprogrammed to identify an open path through vasculature leading into apatient's heart, in a model, and can insert a spline curve as a modeledrepresentation along a centerline of the path. A geometric construct,such as a two-dimensional circle designed to be about the diameter of aplanned implantable medical device, or a 3D model like that for abendable stent (a flexible cylinder), may be run along the length of thecurve. The system may track the location of the anatomical model and themodel of the device, and determine whether they come into contact at anypoint.

If contact is made, the system may note it, and perform both mechanicaland material analysis on any around the points of contact. For example,for mechanical analysis, a system may cause deformation of tissue tooccur in the model using finite element analysis techniques, and mayalso change a display of the model so as to show the tissue moving inresponse to the stress, and perhaps changing the displayed color of thetissue, where the new colors represent the level of stress or strain onthe tissue at particular points. The system may also deform the devicein response to computed forces from the tissue, and may display thedeformed device graphically in an animation and/or generate and save,and display alphanumeric representations of parameters for the forces onthe device and the tissue and the deformation of each. The system maysimultaneously compute fluid flow around the device using computationalfluid dynamics and a 4D visual model may be supplemented with a visualrepresentation of the flow (e.g., with projected flow lines indicatingdirection and speed of the flow). For material analysis, a system mayidentify electrical and chemical changes caused by the normal operationof anatomical item, and may adjust those changes so as to reflect theeffect of the introduced medical device on the tissue. As one example, amedical device may change the propagation of electrical charge through aheart, and FEA or other techniques may be used to model the electricpropagation. Similarly, when drugs are applied to the tissue, such asvia a drug-eluting medical device, the effect of the drugs on thematerial properties of the tissue may also be calculated and used in amodel to indicate the reaction of tissue to the introduced medicaldevice. In processes like those discussed below, different simulationsmay be performed by selecting a medical device of varying materials,shapes, and/or dimensions (e.g., ever-increasing diameters and/orlengths) and determining mechanical (e.g., stress), electrical, orchemical values for each such simulation. A process may then indicate toan operator which of the simulations was performed with a highest orlowest value for a mechanical, electrical, and/or chemical parameterthat is of interest to a user. Or a user may specify a threshold valuefor any such parameter, and the simulations may be performed todetermine which provided inputs qualify. For example, a physician mayset a maximum stress a medical device may place on tissue when thedevice is to be implanted, and simulations may be performed on multiplesizes of increasingly larger devices until the predetermined thresholdis met. The physician may then employ a device of the last size thatpassed the simulation without exceeding the threshold.

The visualization and manipulation may be used by physicians,researchers, and others to better understand a procedure and to helpothers better understand it. For example, a physician may use suchmodels to simulate a procedure for purposes of practice, to determinewhether a particular medical device will seal well or mount well in apatient, and to identify a size of a device (e.g., a heart valve) thatwill best work with a patient. The physician may ask the particularpatient to witness his interaction with the simulated heart or otheranatomical item so as to educate the patient about the procedure—e.g.,before the procedure has occurred. Similarly, medical device companiesor expert physicians may use such simulations to train other physiciansin new procedures, where, for example, an expert physician may initiallyperform the procedure one or more times on the model, and may theninvite trainee physicians to attempt the procedure. Such simulations mayalso be useful outside the context of data for a particular patient, anda model may represent an average or idealized patient (or multipleidealized patients, such as five different patients have differentrepresentative dimensions and other parameters for their organ), such asduring testing of a new medical device for government approval. Forexample, a typical heart may be modeled using an average of data fromthe imaging of a large number of hearts from many different patients,and such a typical heart may be used by a medical device company todemonstrate to a government licensing agency the feasibility of a newdevice or technique.

In certain implementations, such systems and technique may provide oneor more advantages. For example, researchers and clinicians may betterdesign medical devices to interoperate with a body, or may betterselect, size, or manipulate a particular medical device with respect toa particular patient's body. Such visualization may be used as a “dryrun” for a complicated procedure and/or as an opportunity to explain theprocedure to a patient and/or to train others in performing theprocedure. Where FEA is used in such operations, the compliance of apatient's tissue may be modeled so that the tissue interacts with forcesthat a user (e.g., physician) places on it, such as by deforming inresponse to the introduction of a virtual medical device into a body(e.g., so that a researcher or physician can see how firmly the deviceseats when implanted or can see whether any gaps that could create leaksexist around the device).

In one implementation, a computer-implemented medical visualizationmethod comprises identifying a three-dimensional model of an anatomicalitem of a particular mammal; displaying a moving animation of thethree-dimensional model, the moving animation created from multipleframes from imaging of the anatomical item over a short time period tocapture movement of the anatomical item; displaying one or morenon-moving views of the three-dimensional model while the movinganimation is being displayed; and in response to receiving inputs from auser, changing the displayed moving animation and the one or morenon-moving views automatically in coordination with each other. Themethod may also include displaying, in coordination between the movinganimation and the one or more non-moving views, a medical device beingmoved through the anatomical item. The anatomical item may comprise aheart, and the method can also include displaying forces applied on theanatomical item based on movement of the device through the anatomicalitem, where such displayed forces can be determined by applying finiteelement analysis to a tissue model for the anatomical item. In addition,the method can include automatically determining whether the medicaldevice makes contact with tissue in the anatomical item as the medicaldevice is moved relative to the three-dimensional model of theanatomical item.

In some aspects, the method includes performing computational fluiddynamics analysis for fluid flowing around the medical device in theanatomical item, and displaying to the user information representingresults of the computational fluid dynamics analysis. Also, thethree-dimensional model of the anatomical item can be generated byoperations that comprise: obtaining data that represents imagedtwo-dimensional slices of the anatomical item captured by one or moremedical imaging systems; generating a three-dimensional intermediaterepresentation from points identified in the two-dimensional slices; andgenerating the three-dimensional model from the intermediatethree-dimensional representation. Moreover, the moving animation can becomprised of frames taken by imaging a real version of the anatomicalitem at different closely-related times, so as to show movement of theanatomical item. In addition, a first non-moving view may show theanatomical item during a first imaging session, and a second non-movingview may show the anatomical item during a second imaging session. Also,the first imaging session can be a pre-operative imaging session, andthe second imaging session can be a post-operative imaging session.

The details of one or more embodiments are set forth in the accompanyingdrawings and the description below. Other features and advantages willbe apparent from the description and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a conceptual diagram showing a user interacting with avisualization system.

FIG. 2 is a block diagram of an illustrative visualization system.

FIGS. 3A to 3J are flow charts of example processes for interacting withan anatomical feature displayed on a visualization system.

FIGS. 4A to 4L are representation of a user interface for avisualization system for use with parts of a patient's body, such asorgans.

FIGS. 4M and 4N show a representation of a virtual desktop formanipulating and visualizing items such as anatomical items.

FIGS. 5A and 5B show schematically a computation of distance from edgesof an anatomical item to a 3D spline curve.

FIG. 6 shows an example of a computer device that can be used toimplement the techniques described here.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

This document describes systems and techniques for visualizinganatomical features of animals, such as human hearts and vasculature,and other organs on which medical procedures are frequently performed.The visualization may include generating 3D projections for viewing byusers wearing stereoscopic goggles or other mechanisms for realisticviewing of 3D or animated renderings, and 4D animations that showcycling of an organ, such as an entire heartbeat for a heart, where eachframe of the 4D animation may be generated from data capture atcorresponding relevant times during which the relevant anatomicalfeature was imaged.

Users may interact with the models of the anatomical features in variousmanners. For example, a user can zoom in and out and rotate a model sothat they appear to view it from different angles and differentdistances. They may apply cutting planes to the model, includingmultiple simultaneous non-parallel planes, so that they can see internalcross sections of a model—e.g., to see the inside of vasculature or theinner operation of a heart or lung. They may “draw” on the model such asby having a system add geometric constructs around and through themodel. For example, a system may build a spline curve that passesthrough the vasculature in a model, and into a heart. Such additionalgeometric constructs and other meta data for a model may be stored by asystem in relation to the model, so that when the model is closed andthen subsequently opened on a computer system, the meta data may beimmediately displayed (e.g., as graphical and/or alphanumericannotations on the model) or at least immediately available to a userwithout having to redefine the data (e.g., the route of a spline curvemay be saved, as may parameters for a medical device to be moved alongthe spline curve, so that those features may be preexisting when a useropens the model for subsequent sessions).

In other examples, a system may be able to create and align best-fitregistration to a defined coordinate system with three or more markersin each data set, including the ability to synchronize multiple datasets. For example, a user may select three reference points on a heartto associate the coordinate system, may do the same with another heart,and the system may use such selections to display and move the twohearts together in coordination in a common coordinate system. In otherwords, the two hearts may be locked together for manipulation so thatpanning or rotating of one model causes an equal panning or rotating ofthe other model. In certain instances, one of the models may be 4D andthe other 3D, such as a fixed frame from the 4D animation.

In another example, multiple animated models may be displayed andinteracted with by a user simultaneously. For example, a model of ahealthy organ may be shown in animation adjacent to a model for aparticular patient's organ (e.g., heart or lung). The two models may belocked with each other, both in position as discussed above (e.g., ifone is rotated by a user, the other automatically rotates in the samemanner), and also in the location and timing of their cycling—e.g., sothat models of two hearts begin systole and/or diastole simultaneously(where the animation repeatedly cycles through one heartbeat or a setnumber of heartbeats, and then starts again). A physician could use sucha display of dynamic action of both models to better see abnormalitiesin the patient's organ.

FIG. 1 is a conceptual diagram showing a user interacting with avisualization system 100. In general, the system 100 provides anenvironment by which a user 110 may view a 3D representation 112 of amodel of an anatomical item, such as a human organ in the form of thehuman heart. The 3D representation is generated by projecting acomputer-generated image on a screen 102 under control of computersystem 104. The user 110 may interact with the computer system 104through a touch tablet 106, which may be horizontally-mounted in frontof the screen 102 and may include an additional display of the item,along with a number of menu selections made available to the user 110.For example, one or more lines may be presented on the touch tablet 106,and a user may slide the lines across the touch tablet 106 by touchingthe end of a particular line and dragging in contact with the screen ofthe touch tablet 106. Such dragging may cause a line to intersect a 3Dor 3D representation of the model, and may cause the computer system 104to apply a cutting plane to the model so that the user 110 can see aninterior portion of the model.

Sensors 114, which may be implemented as cameras aimed at user 110 andstereoscopic glasses 108 worn by the user 110, may be employed to trackthe position of the user's head so as to make the 3D presentation of themodel more realistic for the user. For example, the sensors 114 mayrecognize when the user 110 tilts his or her head, or moves laterally infront of the screen 102. Such sensed motion may result in the computer104 adjusting a display of the model 112, such as rotating the displayto the right when the user 110 is determined to have moved to the left,so that the user receives the visual impression that he has movedslightly around the model to the left, and is better able to see downinto part of the model, for example.

Various mechanisms other than touch tablet 106 may also be used toobtain input from the user 110. For example, a stylus 116 may be graspedby the user 110 and its position may be sensed by sensors 114 or othersensing mechanisms. The stylus 116 for example, may have its locationidentified and the system 100 may project a virtual line that looks tothe user 110 as if the line is extending out from the end of the stylus116. The user 110 may move that virtual line into contact with the modeland may make a selection, such as by clicking a button on the side ofthe stylus 116, so as to grab the model 112. Subsequent motion by theuser of the stylus 116 while holding down the button may cause the modelto be moved in space in front of the user 110. For example, if the user110 pulls the stylus 116 toward his face while holding down the button,the model may be caused to get larger so as to seem like it is beingbrought closer to the user's 110 face. As an alternative, if the userrotates his wrist while holding a button on the stylus 116, the modelmay be rotated in a similar direction, so that the user may manipulateit to see other areas of the model 112 that were not previously visibleto the user. Other intuitive gestures may also be employed by the user110 to manipulate the model. For example, the user may use the stylus116 to draw lines, curves, or other geometric constructs in the area ofthe model 112, such as by drawing a line through open areas of a heartto define passages through the heart. Such lines may be presented in acolor that contrasts with that in which the model 112 is beingpresented, so that the user 110 can readily see the lines and othergeometric constructs that the user 110 has added to the overall model.

Yet other mechanisms may also be used for interacting with the system100. For example, the user 110 may wear a glove that serves a similarrole to that of the stylus 116. For example, the user may reach out tograb the model 112 with a glove, and the display of the model 112 maymake it appear to the user 110 as is if the user 110 is actuallygripping the model, so that the user may, for example, rotate the modelleft or right or push the model away from him or toward him. Such inputmechanisms may also be provided with mechanisms for applying hapticfeedback to a user. For example, a stylist 116 or glove may have ahaptic clicking mechanism that is energized when a user is determined tohave gripped a model or to have moved from one area on the model toanother, so as to confirm to the user 110 that the system has registeredsuch actions by the user. A particular implementation may also be seendiscussed in more detail below in FIGS. 5A and 5B.

FIG. 2 is a block diagram of an illustrative visualization system 200.The system 200 may implement components of system 100 in FIG. 1, butparticular structural components of the computer system used to generatemulti-dimensional renderings are shown in more detail here foradditional clarity in explanation.

The system 200, in this example, comprises a computer system 202 thatdrives an imaging system 204 in the form of a standard projectiontelevision system having 3D display capabilities. The result of suchdriving is the display of a 3D model 206 in an area where a user, suchas a clinician or medical researcher or patient, may view the model andmay interact with the model in a variety of manners such as thosediscussed above and below. To permit such display and interaction, thecomputer system 202 includes a number of structures that interact witheach other.

A real-time model 208 is a module in the system 200 that accesses dataconcerning one or more models of anatomical items, such as a heart orother organ of the human or other animal, and performs computations onthe model two is the generate data that is capable of producing anaccurate visual rendering of the model. For example, the real-time model208 may include mechanisms for accepting a definition for a model andfor performing various operations on the model, such as distorting themodel, clipping the model, blending the model with other models, andother similar operations.

Information from the real-time model 208 may be provided to a renderengine 212, which may be implemented as software, hardware, or acombination of the two. For example, the render engine 212 may includeone or more graphics processing units and associated circuitry, such asthat mounted on a commercially-available graphics processing cardprovided in a personal computer or mini-computer. The render engine 212may operate according to standard protocols and definitions for modeling2D, 3D, and animate materials for a 4D display (a 4D representation).For example, the render engine 212 may apply shading and other knowntechniques for generating a photorealistic display of the model 206.

A finite element analysis, computational fluid dynamics analyzer 210 maybe referenced by the real-time model 208 where an anatomical item needsto be provided with compliance, such that stresses applied by a user onthe item result in deformation of the item in computationallypredictable manners, or result in deformation of tissue for theanatomical item. For example, a user of the system 200 may invoke a 3Dmodel of a medical device, such as a stent or similar device, and maymove or have the device moved through the model 206, or placed in aparticular location in the model 206. Such placement or movement maycause the device to contact tissue in the model 206, and the analyzer210 may be used to compute the forces applied by the device on thetissue, and the reaction of the tissue to such forces. For example, amodel of a mesh stent may be expanded against the walls of a model of apatient vasculature, and the color of the vasculature in a display madeby the system 200 may change to highlight where the stresses or strainsare the highest or lowest on the wall, such as by generating a heat mapwhose different colors represent particular degree of stress or strain.

The finite element analysis, computational fluid dynamics analyzer 210may perform both mechanical, electrical, and chemical analysis, and mayinterrelate those analyses with respect to the manner that they mayaffect each other. For example, tissue compliance may be used indetermining mechanical stresses and strains from introduction of amedical device. Electrical activity of the tissue may be modeled for itsown purposes, and also to determine how inherent motion of the tissue(e.g., heart tissue) affects how it will deform or otherwise react tointroduction of a medical device. Similarly, the introduction ofchemicals, such as local or systemic administration of drugs, may affectthe action of the tissue, such as by changing its mechanical complianceor changing the way that electrical propagation occurs through it. Suchintroduction of chemicals may thus also be included in a model thatdetermines how the tissue reacts.

The system 200 may also be programmed to recursively perform multiplesimulations that involve automatically passing simulated medical devicesthrough one or more anatomical items, using the computational andmodeling techniques described here. Such simulations may employ thefinite element analysis, computational fluid dynamics analyzer 210 toidentify results of such simulated actions, including resultsrepresented by mechanical, electrical, and chemical activity. Forexample, increasingly larger sizes of stents may be moved through andplaced in one or more anatomical items, and stresses and other factorsmay be computed as the motion of the device occurs. Thresholds formechanical, electrical, or chemical parameters may be set by a user ofthe system 200 or by another (e.g., a developer of analysis software) todefine acceptable versus unacceptable performance, such as a thresholdthat indicates when mechanical stresses on tissue are too high, or anelectrical propagation threshold—either of which are exceeded, forexample, once the simulated device becomes too large. The system 200 maythen identify the proper device as the last in a progression ofevery-increasing-in-size simulated devices that did not cross athreshold into an area of unacceptability.

The various structures already discussed may depend on one or more typesof data in performing the actions described above and below. Forexample, organ models 214 may represent data for one or more organs of asubject, where the subject may be a patient who has had medical imagingperformed one or more times on their body. The organ models 214 mayinclude data that represents spatial relationships in a portion of theanatomical item, in addition to information about tissue compliance andother similar information for the patient or other subject.

Interaction metadata 216 may represent data that is generated inresponse to the interaction with a patient model by a physician or otheruser. For example, a user may make various measurements of a model andthe locations and values of those measurements may be stored so thatthey may be easily retrieved later by the user. Similarly, a user mayform various geometric constructs, such as lines, arcs and otherconstructs in relation to the model 206 in space. For example, a usermay draw a line or spline through passages in an organ or otheranatomical item. The system may also create such constructsautomatically, such as by a user pointing to a passage through whichthey would like a spline to be drawn, and then the system identifyingthe boundaries of the passage and constructing a spline roughly throughthe center of the passage—e.g., computing the center of the passage at aplurality of different locations, and then fitting a spline curve to theplurality of center points that are so identified. Similarly, a user mayinsert particular CAD blocks (CAD representations that include multiplegeometric constructs that are tied together and that represent acomplex, real-world object) in an organ or other structure, such asblocks that represent medical devices to be implemented in an anatomicalitem. Definitions for those blocks, and locations and orientations ofthe blocks relative to the model 206 may likewise be saved in theinteraction metadata 216, so that a user may readily show the blocks inposition in the model 206 during a later session with the system 200.

Physics data 218 may include data that is necessary for representingphysical interaction by a user with the model 206. For example, tissuecompliance information may be stored in the physics data 218 for typicaltypes of tissue, such as vasculature and cardiac tissue in variousstates of deterioration or patient age. Thus, where finite elementanalysis is employed for system 200, particular tissue values from thephysics data 218 may be used, rather than values that may be specific tothe particular patient Similarly, the physics data 218 may include datafor modeling the effect of electrical activity and of chemical activity.

The computer system 202 may also obtain data from external sources vianetwork 220, which may include a local area network, and portions of theInternet. For example, the computer system may obtain imaging data for aparticular patient by accessing a cloud-based storage system where thecompany that did the imaging stores its data for access by users who canprovide proper credentials for accessing the data.

In this manner, then, the system 200 may provide for various mechanismsfor displaying models alone or together with other models, and forvarious users to interact with the models. The system 200 may allowusers to better visualize, and thus be better able to understand,relatively complex physiological interactions, and to appreciate actionsthat occur inside a body and outside of normal human sight.

FIGS. 3A to 3J are flow charts of example processes for interacting withan anatomical feature displayed on a visualization system. In general,the processes may be performed using a system like system 100 in FIG. 1and/or system 200 in FIG. 2, in addition to the systems and userinterfaces represented in FIGS. 4A to 4L. These are example processesthat involve user interaction with a multi-dimensional computer-basedvisualization system. In general, the processes will be described asbeing carried out on a human heart of a particular person being treatedor analyzed by a physician, but it should be understood that otherphysiological items may be modeled and presented, including fromnon-human animals (e.g., for porcine hearts that are part of engineeringdevelopment studies for a medical device), and for analysis by a varietyof users and for a variety of purposes (e.g., physician practice,medical device sizing for a particular patient, education, regulatoryapproval, etc.).

FIG. 3A is a flow chart of a process for identifying and using a paththrough an anatomical item. In general, the process involves a systemdisplaying a view of a model, and the system in cooperation with a userdefining a path through a part of the model for various purposes.

The process begins a box 300, where a patient is imaged, and imagingdata is produced for that patient. Such imaging may occur, for example,using multiple different imaging modalities, including MRI and CT scanimaging. The data produced from such imaging may be in a digital formand may represent slices at different elevations through a part of thepatient. Lines in such images may be used to identify the boundaries ofanatomical items and other items captured in the images.

The generation of a 3D model may occur by a variety of mechanisms,including iso-surface extraction. The iso-surface extraction may beinteractive by a user of a visualization system. For example, suchextraction may include multiple steps that involve generating andrefining a 3D surface using a touch based 2D interface coupled with a 3Ddisplay. In a first step, a computer system may generate a 3D surfacefrom a stack of imaged slices, where each slice comprising a medicalimage acquired from an imaging modality (e.g., CT, MRI, etc.). In a nextstep, the system may render the 3D surface on a 3D display using aniso-surface extraction run on a computer graphics processing unit (GPU)to create an interface to allow a user to virtually touch avisualization of a slice. In such an example, the iso-surface extractioncan be based on iso-values derived from the stack of slices. In responseto receiving user input interacting with a slice from a user operatingthe interface, the system may calculate a refined iso-value used for theiso-surface computation that is based on the input. The system may thenrefining the 3D surface based on the refined iso-value for theiso-surface. Such a process may loop continuously as a user interactswith the displayed model via the interface at box 302, 3D and 4D modelsof the anatomical items captured in the imaging are produced. Suchproduction of models may occur, for example, by identifying lines in theimages of particular slices and placing dots or vertices for a mesh atspaced locations along such lines where the image changes intensity orcolor. The vertices may then be connected to form a mesh, and the meshmay be changed into a solid model that represents the anatomical item.Certain areas of the model may be identified as being filled, in otherwords, being locations where there is tissue from the anatomical item.Other locations may be identified as voids, such as chambers in a heartor through vasculature, or similar open spaces in a patient.

At box 304, user input is received to identify an open path in theanatomical item. Such input may include the user drawing a path througha cross-section of the anatomical item that is displayed on a computersystem. Alternatively, the input may occur by the user instructing thecomputer system to find a best path through the item automatically, andthe computer system identifying the voids in the model and fitting acurve through such voids from a first defined location to a seconddefined location, such as through an aorta and into one or more chambersof the heart. In some implementations, a number of common routes thatphysicians use often may be presented to a user in a list, the user mayselect one such path (e.g., a path for delivery of a stent or a heartocclusion device), and the system may automatically generate and displaya spline along that path for the particular model—e.g., by identifyingopen space in the particular model whose size and shape roughly fits atemplate for such portion of the anatomical item (e.g., particulardiameter and particular curvature, such as for an aorta and heartchamber).

At box 306, the system automatically identifies the path through theitem and fits a curve to that path. For example, an initial path may beidentified, and points along the path at a center point of the void maybe identified at a plurality of spaced-apart positions along the path. Ageometric CAD tool may then be used to fit a curve in a best fit mannerto those points. The system may also confirm, after the curve has beenformed, that the curve does not pass through any tissue in theanatomical item. Other mechanisms may similarly be used to identify acurve through the path that stay adequately within the voids in theanatomical item. Particular details for making such determinations arediscussed in more detail below with respect to FIGS. 5A and 5B, and maybe incorporated into the process described here.

At box 308, the curve is displayed along with the anatomical item. Forexample, the anatomical item may be displayed in various shades of redto represent tissue in a realistic manner, and the curve may berepresented as a blue or green line that visually contrasts with thecolor of the tissue. The curve may be defined in the same 3D space as isthe model, so that a user who manipulates the model, such as to rotatethemselves around the model, may also rotate themselves around the curvein a similar manner (i.e., the curve moves with the model and stays inthe same 3D position with respect to the model).

At box 309, a device is displayed traveling along the path. The devicemay be as simple as a circle having a defined diameter (provided fromthe user) being slid along the curve. A more complex device, such as ablock that represents a 3D model of a stent or other medical device maybe moved along the curve and may bend or otherwise deform as it is movedso as to approximate the shape of a real stent being introduced into apatient. Such display of the device may be used, for example, by aphysician who may select different diameters for a stent or similarimplantable device, and may simulate the insertion of the differentdiameters into the patient by way of the model (e.g., to determine whichsize of a medical device should be selected for a particular procedure).A physician may thus more readily see which size of device may work thebest for a patient, and may avoid trial-and-error that may haveotherwise been required in performing the actual procedure on thepatient.

The device may also be defined to have certain mechanical, electrical,and chemical characteristics. For example, the device may have certainstress/strain characteristics, such that bending it along a splinerequires extra force. Similarly, the device may be coated with adrug-eluting material, and the behavior of such material may be modeled,so that a simulation of moving the device through an anatomical item mayalso indicate a level of drug that is passed from the device to theitem.

In addition, the thing that is moved through and along the spline maysimply be a virtual camera that a user of a visualization system maymanipulate using the hardware and software described above and below.The camera may be anchored to the spline so as to simplify user input,such as by employed motion of one hand of a user to pan the cameraforward and backward along the spline, and employing motion of theuser's other hand to accomplish rotation of the field of view of thecamera.

Where the model of the anatomical item is moving in time (afour-dimensional model), such as a beating heart, the device and splinemay be moved to maintain their positions relative to the anatomical itemin time (e.g., so that the spline and device stay positioned in an openarea, such as a heart chamber or vasculature). Also, such motions maychange mechanical, electrical, and chemical parameters that are computedfor the model, as are described above and below.

FIG. 3B is a flow chart of a process for identifying forces and actionsaround a model of an anatomical item. In general, the process involvesidentifying a model of an anatomical item, identifying forces appliedagainst the model, such as by identifying a medical device that isintroduced into the anatomical item and interacts with surfaces in theanatomical item, and modeling the reaction of tissue in the anatomicalitem using finite element analysis techniques. Such interaction may thenbe represented to a user of a computer visualization system, such as byshowing colored heat maps, where different colors represent relativelygreater or lesser degrees of stress or strain on the anatomical item.

The process begins a box 310, where a patient is imaged in a healthcarefacility and imaging data for the patient is produced. Such productionof imaging data may occur in a manner like that discussed for FIG. 3A.At box 312, 3D and 4D models of the imaged anatomical items for thepatient are produced. For example, a 3D model may be prepared asdescribed above, while a 4D model may be prepared in a similar manner,but may exist as multiple different models presented in sequence overtime, where each of the multiple different models represents a separateimaging of the patient over that time period, where the time period isrelatively short, and obtaining a single imaging session. For example,the time period may be the period of a single heartbeat, such as on theorder of one second.

At box 314, a user input is received, and force applied to the tissue ofthe anatomical items is identified. Standard finite element analysistechniques may be used, such as by identifying interference between amoving model of a medical device in a passage of an anatomical item, andthe distance that the moving device displaces tissue in the anatomicalitem. The device itself may also be compliant such that it is distortedin response to pressure from the tissue as it is moved through theanatomical item.

At box 316, the stress and strain on the tissue and on the device may becomputed using finite element analysis. Such computations may beperformed in conventional manners using data about the tissue andmedical device compliance, as applied to information from the modelshowing degree of movement in the tissue and the device.

In addition to basic finite element analysis on the tissue as a staticitem whose properties do not change, other analysis may also beincorporated, including electrical and chemical analysis. For example,the propagation of electrical energy through a heart as tissue may bemodeled (e.g., using finite element analysis), and may be used to affectthe compliance of the tissue and the forces exerted on and by thetissue. For example, as the heart is a muscle, it may generate its ownforces against a medical device, and such forces change in a cycle everysecond. From such activity, a dynamic model may be generated for asystem by which electrical propagation is used to determine musclecontraction and to then model the forces from and on the heart and fromand on the medical device.

At box 318, fluid flow factors in the anatomical item and around themedical device may be computed using computational fluid dynamics. Forexample, the amount and timing (because the heart is constantlychanging) of blood flow pressure in human vasculature may be modeled,and may be adjusted to match the expected flow for the particularpatient (e.g., by determining via blood pressure measurement and othermeasurements, the flow at a particular location and the strength of thepatient's cardiovascular system). The open cross-sectional area atparticular points along the modeled path may then be determined, such asaround the perimeter of a medical device inserted into the modeledvasculature. From such data, a flow computation may be performed. Thecomputation may then be updated for multiple time periods during acycling, such as during a heartbeat cycle.

At box 319, the anatomical item or items are displayed with visualrepresentations of the computed values. For example, areas of greaterstress or strain on an anatomical item may be shown in colors that comecloser to a red end of a spectrum, whereas lower levels of stress orstrain may be shown by colors that come closer to a blue end of thecolor spectrum. Similarly, in a void of an anatomical item where fluidis flowing, such as in a chamber of the heart, flow lines may bedisplayed either in a 3D representation or an animated 4Drepresentation, showing the direction and magnitude of flow forparticular locations, much like meteorologists display flow lines on amap to show direction and magnitude of wind at particular geographiclocations.

FIG. 3C is a flow chart of a process for coordinating displays ofmultiple models. In general, the process involves displaying tworepresentations of a common anatomical item simultaneously so as toimprove the ability of a user to visualize what is occurring with themodel in interaction with the user.

At box 320, a patient is imaged and imaging data is produced, such as inthe multi-modality imaging and modeling described above. At box 322, 3Dand 4D models are produced such as in manners discussed above, from thedata that is created from the imaging process.

At box 324, a computer-based visualization system displays a movinganimation of the anatomical item or items while concurrently displayingone or more nonmoving views of the item or items. The nonmoving viewsmay be 3D models that are not animated, and are viewed from particularangles and perhaps with particular cutting planes applied, or slicesfrom the imaging data presented in two dimensions. This combination ofdisplay of a 4D representation with a concurrent display of a 3Drepresentation may allow a user to achieve both a dynamic and staticview essentially simultaneously. For example, the user may review theanimated representation of the model to obtain an understanding of howthe anatomical item works in general, and to find a hint of issues withthe anatomical item, and then may quickly switch his focus to one of thenonmoving versions or views of the model so as to focus in on a part ofthe model and more readily sense it in the representation that is notmoving. The user may also step through frames for the 3D model (whilethe 4D model continues to cycle) to get to a point in time that is ofmost interest, such as when an aortic valve is fully closed. The usermay likewise pause the 4D representation, and may have the system copythe 3D representation that is part of that paused presentation, so thatthe copied representation replaces a 3D representation that hadpreviously been displayed next to the animated 4D representation.

At box 326, a system receives user interaction with the model, such as auser introducing a virtual medical device into a model of an anatomicalitem. At box 328, the system computes forces on tissue of the anatomicalitem, such as in manners described above and below. At box 329, adisplay is changed with respect to both the 4D representation and the 3Drepresentation for the anatomical item. For example, if a device like atissue valve or mechanical valve is inserted into a heart, the animationmay change so as to show the opening and closing of the valve and themovement of the heart tissue around the perimeter of the valve. Inaddition, a nonmoving representation may change to show the valve seatedin position, and may also show coloring on the tissue to representstrains imposed on the tissue at particular locations by the presence ofthe valve in its implanted state.

FIG. 3D is a flow chart of a process for coordinating displays ofmultiple models. In general, the process involves concurrentlydisplaying models of an anatomical item, where the anatomical item wasimaged in two different sessions for a patient. In one typical example,one of the sessions may have been pre-operation, and another may havebeen post operation, so that the comparison allows a physician to seehow the anatomical item has reacted to the procedure, and to determinefollow-up that may be required with the patient.

At box 330, the patient is imaged in two different sessions at differentpoints in time (e.g., more than a day apart) and imaging data isproduced from the imaging process such as in manners discussed above.

At box 332, a model for the first imaging session is displayed on acomputer-based visualization system, and at box 334, a model isdisplayed concurrently for imaging from the second session. For example,a pre-operative model may be displayed on the left of a screen, and apost-operative model may be displayed on the right.

At box 336, user input to manipulate one or both of the models isreceived. For example a user wearing a haptic glove may reach out tograb one of the models so as to rotate it. In response to such userinput, at box 338, the system determines a change for both models fromthe single user input. For example, if the user grabbed a right-handrepresentation of a human heart and rotated it 90 degrees clockwise, thesystem may determine that both displayed models of the heart should berotated 90° clockwise.

At box 339, both models are displayed responsive to the user input asjust discussed. Using this process then, a physician or other user mayreadily display a patient organ or other item as it was in existence attwo different points in time, and may readily interact with thosedisplayed models of the item so as to determine how the item has changedbetween the two points in time—e.g., by worsening or improving inphysiology or by having a medical device implanted. As another example,the item may include a part of the patient body with a tumor in it and aphysician may manipulate the area around the tumor, such as by panningaround the tumor visually, to better appreciate how the tumor has grownor shrunk, and to better determine what sort of action should be takenwith respect to the tumor.

FIG. 3E is a flow chart of a process for providing measurements of amodel. In general, the process involves a user making selections in a 3Ddisplay or 4D display of a model of an anatomical item, and a systemmaking various measurements based on those selections.

At box 340, a patient undergoes medical imaging and imaging data isproduced from such a process, such as in the manners discussed above. Atbox 342, 3D and 4D models of anatomical items captured in the imagingare produced. At box 344, user input is received at a computer-basedvisualization system, where the user input defines locations proximateto the displayed models. For example, a user may employ a stylus topoint a virtual line at one wall of a vessel and then an opposite wallof the vessel. The system may be programmed to understand that suchopposed locations indicate a desire by the user to determine a diameterof the passage at the selected locations or near the selected locations.The model may then be interrogated to identify the distance in two orthree dimensions between the selected points, after identifying pointsthat are adjacent to those selected by the user by that define a lineperpendicular to a line through the passage (so that the diametermeasurement uses too directly opposed points even if the user did notpoint at directly opposed positions). Such distances may correspond tothe actual distance within the anatomical item in the real-world, suchas being on the order of less than an inch even when the model appearsto the viewer to be several feet across, because the viewer has zoomedin on the model as part of the visualization session. Other similar 2Dor 3D or 4D measurements may also be made, including computing thevolume of a chamber of a heart, computing the cross-sectional area ofvasculature and other measurements that may be desired by a physician orother user.

At box 348, the model is displayed with data that indicate thedimensions that have been determined. Such display may occur in multipledifferent manners. For example, an arrow may be drawn across thediameter of the passage that the user has selected, and data may beoverlaid on the arrow representing, for example the diameter the passagein millimeters, centimeters, or inches. Separately, an annotation boxmay be displayed elsewhere in a visualization representation, away fromthe display of the model itself. For example, a small spreadsheet may begenerated and displayed that represents the measurements computed by thesystem in response to the user input—e.g., diameter measurements every xmillimeters along a user-defined path.

At box 349, additional representations may be made within avisualization system where a model is animated. For example, where auser attempts to determine a diameter of the passage, the diameter maybe constantly changing in an animated model. As such, the system may beprogrammed to identify the diameter at every frame in the animation andto provide feedback to a user consistent with such determination. Forexample, a selection may be made in a 4D model, and the system maygenerate a 3D model adjacent to the 4D model, where the 3D model isannotated with dimensions. Such annotations may include two numbers foreach such dimension, where one number is a maximum and another number isa minimum during a cycle represented by the animation, such as a singlecycle of a beating heart.

FIG. 3F is a flow chart of a process for generating a solid model of ananatomical item. In general, the process involves capturing data frommultiple imaging modalities, correlating and analyzing the data, andthen building a multiple-dimensional model out of such coordinated datafrom multiple imaging modalities.

The process begins a box 350, where a patient is imaged using multiplemodalities, where a modality is a particular style for imaging. Typicalsuch modalities include magnetic resonance imaging (MRI) and computertomography (CT) imaging. Multiple modalities may be used because certainmodalities may have particular strengths relative to other imagingmodalities.

At box 352, data from the multiple modalities is coordinated. Forexample, images from the data may be analyzed to identify locations inan anatomical item that are the same as between two different imagingmodalities. Such analysis may include identifying an outline of ananatomical item and matching it to an image from another modality havinga similar outline. Such matching may be simplified if one knows from theimaging data the direction at which the particular anatomical item wasimaged and matching it as between the two modalities, i.e., the item wasimaged from the same direction for both modalities.

At box 354, a mesh of points is created by layering image data in athird dimension. For example, slices may be analyzed in the order thatthey were stacked for the actual anatomical item, and nodes for a meshmay be identified along boundaries in each of the slices. The nodes maythen be stacked in a vertical dimension and may together form a meshthat substantially defines the anatomical item.

At box 356, a solid model is created from the mesh, and at box 358, amodel is displayed that has been created from the multiple imagingmodalities. Such generation of a solid model form a mesh may occuraccording to a number of known manners.

FIG. 3G is a flow chart of a process for coordinating metadata with amodel of an anatomical item. In general, the process involves savingmetadata along with a 3D or 4D model. The process begins a box 360,where a patient is imaged, and imaging data is produced from suchimaging.

At box 362, 3D and 4D models of anatomical items from the imaging areproduced and displayed. At box 364, a user input is received to identifyitems to display with the model. Such items are separate from theanatomical item, and may include geometric constructs and medicaldevices to be introduced into the model of the anatomical item and tothus annotate the model in a visual representation for a user.

At box 366, data about such extra items may be saved in association withthe model and with a common spatial frame of reference. In particular, afile where the model is saved may reference a file where the additionaldata is saved, so that when the model is reopened at a later date, theadditional items can also be opened with the model. The common frame ofreference allows the items to be located at their same location relativeto the model. For example, a spline through an open path in an organ maybe reset in its proper location the next time the model for the organ isopened.

At box 368, a user selection is received to display the model. In thisexample, the user selection is made during a later session, perhapsafter the user has shut down the visualization system or looked at othermodels with the visualization system. Because the additional data wassaved in the manner defined at box 366, the model may be displayed withthe additional items when it is later opened, as shown at box 369.

FIG. 3H is a flow chart of a process for cutting a model. In general,the process involves displaying a model that is being cut along multipleplanes so as to provide a potentially complex view into an interiorportion of the model.

The process begins at box 370, where a patient is imaged, and imagingdata is produced for that patient. At box 372, a 3D and 4D model of ananatomical item or items that were captured by the imaging are produced.At box 374, one or more views of the model may be displayed. Forexample, an animated 4D view may be displayed adjacent to a non-animated3D view of the model. In this example, both views may simply show theouter surface of the heart, and a user may want to get a better view ofthe inner portion of the heart. A single cutting plane may do aninsufficient job of providing such a view because walls of the area mayblock a complete view of what a user would like to see inside the model.

As a result, at box 376 the user provides input that definesmultidimensional cutting planes. For example, a patient heart may bedisplayed in a visualization system and a user may slide multiplerepresentations over the displayed model of the heart so as to definethree cutting planes that cut a corner out of the heart. At box 378, themodel is displayed with the portion that was defined by the cuttingplanes removed. For example, if the user moved a cutting plane over acorner of an atrium of a heart, subsequent display of the model of theheart may show a part of that corner of the atrium removed. If multipledifferent models of the heart are shown (e.g., from imaging sessionsthat occurred at different times for the patient), a set of cuttingplanes can be applied to the second representation automatically inresponse to a user applying them to the first representation.

FIG. 3I is a flow chart of a process for building one or more models ofan anatomical item. In general, the process involves breaking ananatomical item into sub portions and allowing a user to readilyinteract with those sub portions. For example, the parts of an organsuch as the chambers of the heart may be identified when a model of theheart is built, so that a user may readily change colors up the chambersindividually, or change other viewable properties of those chambers.Effectively, the portions of the heart are represented as layers thatcan be manipulated as individual groups, such as to turn them on and offvisually, to color them, and the like.

The process begins at box 380, where a patient is imaged, and image datais gathered. At box 382, portions of an anatomical item from the imagingdata are identified. For example, a system may be programmed to identifywalls surrounding chambers in a heart from slices in the data, and mayassign particular nodes in a mesh generated from the slices to layersfor one or more of the chambers. As the mesh is defined in threedimensions, additional ones of the vertices in the mash may be definedaccording to a chamber to which they correspond, as shown at box 384.

At box 386, a 3D solid model is generated from the vertices in the mesh,and may inherit the portion identifiers that were assigned to thevertices that correspond to those parts of the solid model. For example,zones in the solid model may be identified and associated withparticular ones of the layers.

At box 388, the solid model is displayed. Such display may includevisually identifying the different portions of the anatomical item thatwere identified in the earlier steps, such as by providing differentcoloring or shading to various chambers of the heart. At box 389, userinput regarding particular layers in the model is received, such as auser wishing to see only the heart cell layers to the exclusion of allother layers in the heart. The concept of layers in this example issimilar to that of layers used in CAD and GIS applications, wherecertain layers can be turned on or off, or may be assigned propertiesaccording to all items that are identified as being in a particularlayer such as a color property that changes all items in a layer to theassigned color.

FIG. 3J is a flow chart of a process for providing visualization of ananatomical model. In general, the process involves providing a displayof a model of an anatomical item while a user views the actualreal-world representation of the anatomical item. In one typicalexample, a surgeon may be shown a model of a heart or other organ of aparticular patient while performing surgery on that organ of thepatient. In certain examples, the displays may be superimposed over thenormal vision of the user, such as using hardware similar to thatrepresented by the GOOGLE GLASS hardware, supplemented with anapplication to obtain imaging data from a hospital information systemand to respond to physician voice commands for manipulating the modelthat is displayed.

The process begins at box 390, where a patient is imaged, and imagingdata is produced from such actions, such as in the manners discussedabove. At box 392, 3D and 4D models of one or more anatomical items inthe imaging are produced. At box 394, input is received from a user whois viewing a procedure so as to display the model. For example, a 3Dmodel of a human heart may be displayed that is cut vertically down themiddle so that a surgeon can see the inside of the heart in the model.The heart that is displayed may be a model generated from imaging of thepatient who is in front of the surgeon at that time.

At box 396, user interaction with the patient may be identified and adisplay of the model may be updated accordingly. For example, a patientvasculature and heart may be imaged and modeled in preparation for anangioplasty or some similar procedure. The depth of insertion of asurgical item, such as an angioplasty balloon may be monitored fromoutside the patient and such depth of insertion may be supplied to avisualization computer system that is displaying the model. A separatemodel of the device may then be displayed by such a system in relationto the model of the vasculature and heart, so that the physician mayview, out of a corner of his eye and on a heads-up display, the relativeprogress of the medical device in reaching its ultimate destination.Such location information may be confirmed using other standard imagingtechniques that are employed during a procedure, such as tracking aninstrumented portion of the medical device using imaging technologiesoutside the patient.

At box 398, a display of the model is annotated with real-timeinformation. For example, a system may display a numeral representing adepth of insertion of a medical device into a patient (e.g. in inches).Similarly, as a physician introduces the medical device and interactswith walls of the heart, similar interaction may be simulated in themodel, and annotations may be provided on the heads-up display toindicate to the physician the level of stress or strain that is beingapplied at that moment to the tissue. Other annotations may also beprovided to assist the physician, such as heart rate and blood pressurethat is currently being measured for the patient. In addition, notesfrom assisting staff may be displayed to the physician using anotifications interface that is provided with an operating system forthe heads-up display.

At box 399, a record of a video of the procedure is saved along with thesuperimposed model view so that a representation of what the physiciansaw during the procedure may be maintained. Such a recording may beprovided for training to other physicians, may be studied by thephysician himself to improve his technique, or may be used in multipleother manners. The video may show a camera view from the front of a pairof glasses, such as GOOGLE GLASS, worn by the physician, and in onecorner of the recorded video, a display may be shown representing whatthe physician was presented by the visualization system at the same timeas the physician was viewing the procedure (e.g., the model of the organand/or alphanumeric and graphical annotation data).

Various processes have thus been described to indicate uses for animaging system that presents models of anatomical items, like a heartand vasculature, in 3D and 4D presentations. Such processes may be usedby physicians during procedure, by physicians in preparing for aprocedure (e.g., to identify a proper size for a medical device to beused with a particular patient), by physicians in explaining to apatient his or her pending procedure, by a physician or salesperson totrain another physician or to explain the benefits of a particulardevice or procedure to the physician, by medical device manufacturers atconferences to explain new procedures, and for a variety of othergeneral uses.

The following discussion addresses more generally virtual prototypingtools for use in medical device interactions, including a user interfacefor a computer-based visualization system that generates views ofmedical devices in relation to parts of a human body and allows a userto manipulate such virtual devices and receive both image-based andalphanumeric data-based feedback about such manipulation, such as byseeing a “heat map” display that changes the color of displayed tissueas a function of forces exerted on the virtual tissue by the virtualmedical device, in addition to seeing a table that provides alphanumericdescriptions of those forces (as just one example).

The system may include a 3D touch table VR holographic imaging system ora commercially available VR system, such as a system from zSpace ofSunnyvale, Calif., which may be provided with additional programming tocarry out the operations described above and below. Features of such asystem may include: (a) integrated file loading for image sequences andtriangles meshes; (b) support for loading and interacting with multiplemeshes; (c) support for loading multiple meshes inside of a singlelinked coordinate frame; (d) support for multiple splines insideenvironments; (e) support for loading and saving splines to disk; (f)customization tools for changing the color of meshes/splines; (g) camerapath support for flying the view windows along splines; (h) videorecording functionality that supports external stereo viewing on a 3DTV; (i) real time iso surface extraction from loaded image sequences;(j) project loading/saving for saving the entire state of the program;(k) view resetting if meshes get lost; (l) device sizing option forsetting a radius of a device to understand how it fits in the geometry;(m) interactive visual feedback and widgets that make the interfaceintuitive, even for novice users.

Additional functions that such a programmed and implemented system mayinclude are: (a) loading FEA data into both systems; (b) per-vertexcoloring of meshes; (c) three views (sagittal, coronal, transverse) inthe upper left corner of the projector screen that are stationary and donot scale with user controlled zooming; (d) viewing andcontrolling/manipulating 4D anatomical data sets; (e) ability to rotatecutting planes; (f) ability to toggle cutting planes on off; (g) abilityto toggle bounding boxes on and off; (h) measuring capabilities, such asaverage distance to center of a spline (e.g., to tissue surrounding thespline in a representation of a spline in a human organ) and centerpoints relative to a mesh.

To that end, the following figures discuss software and hardware used inone example of an immersive multi-touch workbench setup, and provide anoverview of one configuration and example settings for various systemcomponents. The description discusses selected system components andprovides procedures for configuring the system. A first sectiondescribes the general system setup, detailing the operating systemsettings. A second section details the camera hardware and software totrack the user's head position. A third section describes themulti-touch overlay that is used to provide touch input to the system. Afourth section includes a description of the custom model viewersoftware.

General System Setup:

Different examples of the present subject matter can include varioustouch hardware setups. One example setup includes a projector, andvarious setups can include two screens or a third screen (monitor). Thedescription here can serve as a guide, and individual configurations maybenefit from customized tuning. This section describes the computer andperipherals along with relevant Windows OS settings (though the systemmay operate using a variety of operating systems). There are three videooutputs in the example system: (a) a projector displaying on a verticalscreen; (b) a large-scale flat monitor (e.g., LCD) displaying on ahorizontal touch surface; and (c) a computer monitor positioned to theside of the other displays and intended for desktop use. Appropriatevideo cards may be used for generating such graphics in a coordinatedmanner

Head Tracking Cameras and Software:

An example of the head tracking subsystem includes four IR camerasmounted above the system, a USB hub connected to each camera, and thesoftware that runs the cameras. This system identifies the location ofthe user's head by tracking reflective markers affixed to the user'shead, such as on a headset worn by the user. In one example, the markersare affixed on the head using eye glasses. The system streams theposition and orientation of the head position to the model viewingsoftware. Accordingly, the software can be running before the start-upof the model viewer software. The start-up procedure for this softwareis executed after it has been calibrated. Additionally, the camerasshould be in a stable position and should not need readjustment.However, over time it is possible that due to the set up being bumped,or the loss of the main configuration file, this system may need to becalibrated.

Multi-Touch Overlay:

A touch overlay sits on top of large horizontal TV display and detectstouch contacts and resends them to a computer in a similar way that thehead tracking data is streamed to the computer. It connects to thecomputer via a USB cable and has a separate cable to power it. Inaddition to sending the touch input info to the model viewer software,the touch overlay also interacts with the operating system, which is whyicons and cursors pop up when touching the surface even when not runningthe program. In this way, the input can also act like a mouse for thesystem. The options for the overlay software can be accessed through thetool tray menu in the lower right of the screen as a “cross-hair” icon,as shown in FIG. 4A.

Model Viewer Software:

In one example, model viewer software supports loading of two differenttypes of data (that are typically linked to each other): (a) 3D trianglemeshes (typically a Standard Tessellation Language {STL} file format);and (b) sequences of medical imaging files that define volumes(DICOM/png image stacks). The system supports loading an arbitrarynumber of image stacks and/or triangle meshes into the software. Whenthe software opens, a default set of data is loaded. New data can beloaded to replace these existing sets. The system works by supportingmultiple “frames”, where each frame can contain multiple trianglesmeshes and a single linked image stack (i.e. volume). These are called“frames” because all the data loaded in them shares a single coordinateframe origin. Thus, when manipulating this frame, all the containedmeshes and the image stack are manipulated together. Each of theseframes is visually depicted by the red bounding box that shows itsextents. FIG. 4B shows an example of such multiple frames, indicated byred bounding boxes, each containing a 3D heart geometry.

The model viewer software is effective when the frames contain bothtriangles meshes and imaging stacks together. However, for this to work,these two sources of data are to be registered so that they line upproperly. The software accomplishes this by making certain assumptionsabout the coordinate frames of the triangles meshes, and conventions inhow the image stacks are mapped into 3D. The following paragraphsoutline these assumptions and conventions. There are multiple ways tomeet these conventions. For example, in the export process of thesoftware being used to segment the geometries, they can be keptconsistent. However, with existing data that does not meet theassumptions, to load the data, a piece of software that is helpful inchanging coordinate frames is the free 3D triangle mesh software calledMeshLab.

The global coordinate frame for the entire virtual environment isdefined by a general Cartesian coordinate system with the positiveY-Axis points upward. When triangles meshes are loaded in, they willdefault to match this orientation (their local frames can be rotatedafterwards though), as shown in FIG. 4C. That figure shows the order inwhich the series of Dicom slices are loaded, the first image (here shownas 000) as being loaded at the bottom of the volume, and the last image(here labeled XXX) loaded at the top. This order should be maintained toproperly orient the volume with the mesh. An image stack is assumed tostack in the positive Y-Axis direction. Thus, the positive Y-directionof a loaded mesh should match the direction associated with increasingsequences in the image stacks.

To register meshes and image stacks, there are two distinct approachesthe software can use. The first approach (basic) is used when theextents (i.e. bounding boxes) of the mesh and of the image stacks areexactly the same. The second approach (advanced) is used when theextents do not match up and additional information must be input toregister the data properly. These two methods are distinguished by theorder in which the data is loaded. If triangle meshes are loaded first,and then an image stack is prescribed to it, the basic method is used.If the image stack is loaded first, then the more advanced method isused to set up the volume properly, after which a mesh can be loaded inthe registered coordinate frame. These two processes are detailed in thefigures below.

FIG. 4D shows the basic loading process. The mesh is defined first (inthe left image)—its local coordinate origin can be essentially anywhereand its extents will be indicated by the red bounding box. After a meshhas been loaded, an image stack can be applied (shown in right image)and will fit to match the extents exactly. This strategy will only workif the meshes extents exactly match the loaded volume's extents.

FIG. 4E shows the advanced loading process. The volume is loaded—itslocal coordinate origin is fixed to be in the back-lower-left (minimum)corner of the bounding box (which must be defined via additional input).Additional meshes can be loaded and if they share this common coordinateframe, they will be properly registered. This technique is helpful ifonly a subset of the image stack was segmented.

Program Controls (Touch Hardware):

The model viewer software uses a World-In-Miniature (WIM) metaphor tonavigate and interact with the world. The WIM serves as a smaller map ofthe world that floats above the table surface, as seen in the figurebelow. The current position inside the world is indicated relative tothe WIM by the “View Window.” The larger scale version of this view isshown in the “Detailed World” behind the WIM. Finally, two widgets areprojected on the table surface that the user interacts with. First, the“Shadow Widget” is a vertical projection of the WIM down onto the table.Second, the “Slice Widget” is a vertical projection of the view windowonto the table. The metaphor is shown graphically in FIG. 4F. The ShadowWidget is used to directly control the WIM itself, either to positionand scale it to different location or size, or to rotate and orient itto view the world from a different direction. A set of gestures is usedto perform each of these actions, as seen in the figure below. Thesegestures can be enacted anywhere on the table surface (except the SliceWidget). These gestures can be used on individual frames or across allframes by interacting on top of the shadows of objects or elsewhere onthe screen.

FIG. 4G shows (a) gestures for translating, scaling, and rotating theWIM in the plane of the table surface, and (b) gestures for tilting androlling. The Slice Widget can be used to directly control the currentposition and scale of the detailed world. The green line projection onthe table provides a set of handles to grab the ends of the floatingview window. These handles are indicated by the blue circles at the endsof the Slice Widget, as shown below. Each of these handles can beindependently grabbed and manipulated to change the position of the ViewWindow. To change the zoom level, the window can be scaled to be biggeror smaller. The yellow line indicates the current viewing direction intothe detailed world. The vertical height of the View Window can beadjusted by moving the thumb near the double-sided arrow indicated inred. By pressing and holding the rotating arrows in the center of theSlice Widget, the WIM can be spun around by moving a second finger onthe table surface while fixing the View Window in place to flip theworld around to see from the other direction. The lock icon near eachhandle can be used to change the viewing properties of the WIM bytapping the lock icon to cycle through the options. In the default mode,open lock, the WIM will only appear when the user is interacting withit. In the locked mode, indicated by a closed lock, the WIM will alwaysstay visible. In the hidden mode, indicated by an empty circle, the WIMwill always be invisible, even when interacting. A fourth mode willalways hide the zoomed in view and only show the WIM.

FIG. 4H shows a slice widget. Handles (circles on both ends of the line)are used to position the view window inside of the miniature world. Theheight of the horizontal slices can be adjusted up or down by draggingthe thumb near the double arrow in the lower right of the figure.

A feature is available for specifying curves and volumes inside patientanatomies, i.e., in particular anatomical items. Placing a finger downon the center of the Slice Widget and dragging out from it will create a3D point. The point is moved in a plane parallel to the table surface bydragging it around the table. To change the vertical height of the 3Dpoint, a second finger, usually in the opposite hand, is pressed on thetable and a “virtual string” is connected between the user's fingers andthe 3D point floating above the table. Bringing the two fingers togetherloosens the string, allowing the 3D point to go higher. Similarly,moving the two fingers apart lowers the 3D point. Finally, with twofingers pressed in this manner, a third finger, usually a thumb, movedvertically on the table, will inflate or deflate the curve at this pointto encompass a volume. Such a gesture is shown graphically in FIG. 4I.There, points are specified in the 3D space relative to the miniatureworld. Multiple points are combined to form curves and may be expandedvia an additional touch gesture to define a generalized cylinder.

The software described herein can be operated on the zSpace hardwareplatform. The main input device there is 6-degree of freedom wandinstead of a 2D touch input surface. This wand's translation androtation are both tracked, and all the manipulation described in thetouch hardware is replicated by using the wand combined with threebuttons on it. To retain the concept of a “table” without a seconddisplay, the horizontal table surface is moved backwards beyond theangled zSpace display and exists only in virtual space. Additionally,the world-in-miniature visuals and interactions used in the touchenabled hardware setup are removed in the zSpace version. Only theminiature is shown and manipulated by the user, and no view window isused.

FIG. 4J is an illustration of the model viewer software components onthe zSpace hardware. The vertical screen rests at a 60 degree angle froma table on which it sits. The virtual table is drawn behind the windowand the shadows are projected on it. The stylus device, indicated by thecoordinate frame and yellow line, are used to interact with the virtualenvironment. The tracked stylus (or wand) is used as a virtual cursor tointeract with the model viewer software. It can be used to simulate atouch event on the table surface by pointing it down and intersecting aray with the table. When pressed, this button combined with stylusmotion will simulate a touch event on the virtual table. In this way,all single-point interactions used in the touch hardware can beperformed. In addition to this basic capability, other more advanced 3Dinteractions (that would otherwise require more than one touch point)are supported. A large main button on the stylus serves as a “grab”button. By pointing the stylus at the bounding box of each frame andclicking the grab button, the mesh can be moved in a one-to-one mappingfollowing the stylus' position and orientation. There are also otherfeatures that can be manipulated with the grab button; for example, thehorizontal slice can be grabbed and moved up or down to change the viewdepicted in the shadow on the virtual table. There is also a vertical“wing” on the left side of the virtual table that can be moved forward,backward, upward, and downward to change how far away the verticalclipping plane is. The grab button can be used to grab onto the locationof controls points of splines to position them in 3D.

The right button on the stylus serves as a “scale” button. By pointingat a bounding box followed by a clicking and dragging action, the buttoncan be used to scale the data (e.g., meshes or solid models) containedin that bounding box's frame either up or down by moving away or towardthe center of the box. The scale button can also be used in this sameway to change the radius of the tube created by the spline. This scalebutton also has an additional feature that can be used to rotate thevirtual table upward, with a hinge at the front end of the zSpacedisplay. This is useful for interacting more easily with the tablesurface, which is helpful when using the 2D table widgets describedlater in this document. This can be performed by pointing the stylusdown at the table, pressing the scale button, and moving the stylusupward. Another function of this button relates to the “wing” mentionedearlier. By clicking on the wing and pressing this button, the wing canbe rotated to change the angle of the clipping plane so that is it notperfectly vertical.

A third button on the left of the stylus serves as a “cursor” button.Clicking on this button will create a new 3D cursor for a spline in thenearest frame. This works in the same way as the dragging operationdescribed above in the touch hardware. After clicking, the cursor can bepositioned and will remain stationary after the button is released.

FIG. 4K shows a display of a virtual table with drag-able touch widgetsdocked on the right of the screen. Each of these widgets can be movedout onto the table and used to change settings of the virtualenvironment. Many additional features are available to the model viewersoftware, including the ability to load and save files or change thevisual settings of the virtual objects. The software makes use of atouch widget metaphor for implementing these features. A set of widgetsis docked on the right of the screen. Each of these can be dragged outonto the virtual table (via touch or stylus interaction). These widgetscan then be interacted with to perform various actions. Often theseactions are dependent on a widget's location on the virtual table. Forexample, if the widget is located on a specific shadow of a mesh, itwill change the settings for that specific mesh. The following sectionswill go through each of these widgets in more detail, describing whatthey do and how to use them.

In addition to the visual square icon representing the touch widget,each widget can have four different features that are consistent acrossall the widgets in their functionality. These four features are labeledin FIG. 4L. Each of the arrows can be clicked on and will be expanded tocontain more controls. If the widget contains a focus arrow, then thelocation indicated by the tip of the arrow corresponds to either whichdata elements are modified by changing the settings or corresponds towhere the file loaded will be located upon finishing the load operation.

The Program Exiter widget is used to close the program. Simply drag itout onto the table and double click it to exit the software.

The View Resetter widget is used to reset and center the currentlyloaded frame plus the view window. It is useful for when the virtualobjects get lost, typically caused by extraneous inputs on the touchsurface. It will reset the scale of the objects so they fit on thescreen well, and array them horizontally along the center axis of thevirtual table. The Color Picker widget is used to set the color ofindividual meshes inside of the frames. After it has been placed on thetable, it will display a paint brush icon combined with a circular colorpalette. The mesh with its center closest to the tip of the paint brushwill be selected. After selecting, a touch point inside the colorpalette can be used to drag and change the color applied to the mesh.

The Delete Tool widget is used to remove/unload existing frames (i.e.meshes and image stacks) from the software. Simply move the focus arrowon top of the frame to be deleted and double click the widget to deleteit.

The Spline Tool widget is used to modify, load, and save 3D splines(curves) that have been created using the earlier described splinegeneration interactions. The file type used to hold the splineinformation is a comma separated value (csv) file that can easily beread into spreadsheet software. It has two sections. The first sectionlists the locations of the exact control points used to create thespline. The second section lists the points resulting from evenlysampling the mathematical representation of the spline curve. Each ofthese is listed using four columns—the first three columns define the X,Y, and Z position of each point. The fourth column defines the radius ofthe spline's tube at that control point. To save a spline, the widgetcan be dragged so that focus arrow selects a curve on the table. Then,by expanding the save window and selecting a directory, a csv file willbe created with an auto-generated file name time stamped to when thefile was created.

To load a spline from disk, drag the widget into the frame where thespline is to be loaded, expand the load window on the left of thewidget, and select the .csv to load. The spline will be recreated inthis frame and can then be modified. There is also a set of widgetsettings that can be used to change the behavior of splines, andinformation associated with each spline, that can be seen by expandingthe arrow on the bottom of the widget.

A description of these elements is as follows: (a) total SplineLength—Reports the entire length of the currently selected spline; (b)current Segment Length—Reports the length of the distance between thelast control point and the previous control point; (c) Avg Distance toWall—Reports the average distance from the center of the spline to thenearest boundary found radially outward from the spline. This value iscomputed on demand when the button mentioned below is pressed; (d)Frozen (Check Box)—If checked, the spline will be locked into place andcontrols points can no longer be manipulated; (e) Trailing Buffer (CheckBox)—If checked, only the last three controls points of the spline willbe modifiable, acting as a trailing buffer. This is helpful if thespline contains many points, such that they become difficult toindividually select due to clutter; (f) Center Spline (button)—Whenclicked, the entire spline will be centered relative to the surface thatsurrounds it; (g) Compute Average Dist (button)—When clicked, theaverage distance from the spline to the nearest wall will be found bysearching radially along the spline. This value is reported in the abovementioned text box; (h) Generate Radial Spline (button)—This button isused to generate a new spline that follows the exact shape of thesurface. It uses an existing spline to seed the generation of the newspline. The seed spline must intersect the plane of the vertical wallsurface. The point where this spline intersects is used to shoot rays inthe plane of the vertical surface and control points are positioned atthe intersection of these rays. For example, if a spline was placed thatfollowed the center line of an artery, and the view was rotated so thisspline is clipped at the vertical plane, after this button is picked anew spline that lies in the vertical plane will be created that followsthe contour of the inside of the artery.

The Animation Tool widget is used to control the settings for temporaldatasets that have been loaded into the software. This includes both 4Dmeshes and 4D image stacks. After a frame has been loaded with either ofthese types of data, this tool can be dragged onto its shadow and thefollowing information/settings are available: (a) Total AnimationTime—The current time in seconds to complete the entire sequence ofloaded animations, change this setting to match the desired animationspeed; (b) Number of Volumes—Reports the number of volumes in thecurrently loaded volume sequence of the selected frame; (c) AnimationInterval Time—The time step between consecutive volumes in the sequence.Total Animation Time=Number of Volumes*Animation Interval Time; (d) MeshAnimation Time—Sometimes individual mesh sequences inside of the framehave different timing than the other meshes (or the loaded volumesequence). This setting controls the timing for the individual meshsequence that is closest to the focus arrow. By default all loaded meshsequences match the total animation time, however it can be changed bymodifying this value; (e) Number of Meshes—Reports the number of meshesin the mesh sequence that is closest to the focus arrow; (f) MeshAnimation Interval Time—Analogous to the “animation interval time”above, but for the individual mesh sequence; (g) Playback Speed—thespeed at which the animation is played, defaults to one for real timeplayback, but is adjustable to speed up or slow down the animation. Inaddition, this widget has an additional time line window that can bemoved around. This time line has a play/pause button attached to it, tostart or stop the animation of the frame currently selected. The timeline shows an indicator of how far in the animation the current displayis. The time line is interactive, by dragging along it the current timestep can be changed.

The Volume Tool widget is used to configure the dimensions of loadedimage stacks, as well as several controls for changing the visualizationsettings. This tool is used to properly set up the coordinate framelocated at the minimum corner of the image stacks 3D bounding box (asdescribed in section 5.3). As a user changes the input, the bounding boxof the volume will update. It will continually resize so that it doesnot become so large or small it can't be seen anymore. Despite this, thebounds of the box are changing (and a scaling factor is being changed toaccount for this). The bounding box must be set up correctly so thatsubsequent loaded meshes are in the correct position and scale. Thesettings in this section should be configured to match theunits/coordinate frame in which the subsequent meshes to be loaded aredefined. The available settings are:

The Volume Scale to Meters widget is used to set the units that themeshes/volume are using, when multiplied by this scale factor, the unitsshould be in meters. For example, if the meshes and volumes wereexported using mm as their units, then this scale factor should be setto “0.001”. This factor is in turn used to ensure the display outputsare reported in their correct unit (which would be unknown without thisfactor). Various check boxes operate as follows: (a) Texture from Volumeturns on/off 3D texturing on the meshes. If on, the image stack will betreated as a “block of marble” and the colors contained in the volumedefined by the stack will be mapped onto the meshes surface; (b) ShowSolid Slice turns on/off solid modeling for the selected mesh. If on,the mesh will be treated as a solid object that has a defined inside andoutside, when clipped into, the inside will be rendered flat at theclipping plane, such that the inside is hidden from the clipping (i.e.back faces are not visible). This should only be used on watertight,well-defined meshes where the front and back faces are all consistent;(c) Show Shadow Slice turns on/off a shadow representation of thecurrent horizontal slice through the meshes; and (d) Texture WIM turnson/off a 3D texturing on the WIM, a 3D texture has been set.

The Device Sizing Tool widget is used to configure the device sizingtool which is available for measuring out diameters in the virtualenvironment. The red line on the view window's shadow can be moved todirectly control this virtual device radius. In addition to this basiccontrol, this widget allows precise setting of the size. The followingsettings are available: (a) Freeze Device Width (check box)—Whenchecked, the device will be frozen at its width and cannot be modifiedanymore; (b) Always Show Device (check box)—When checked, the circulardevice indicator will always be shown, instead of the default behaviorwhich is only shown when it is interacted with; (C) Never Show Device(check box)—When checked, the circular device indicator will never beshown; (d) Device Width (mm)—Allows the user to precisely set the devicewidth (diameter) in mm.

The Camera Path Tool widget is used to animate a camera (i.e. the viewwindow) along the path of a spline. First, a spline must be created inthe desired environment. Then, this widget can be dragged onto theshadow of a spline. After positioned on a spline, the camera can becontrolled using an additional set of buttons. The left-most button (theright angle) is used to square up the view window so that it isperpendicular to the direction of the spline. After square up, the“play” and “reverse” button can be pressed to start the camera along thepath in either direction. If the opposite one is pressed when it isanimating, the camera will pause. Additionally, the right most buttonscan be pressed to either speed up or slow down the motion of the camera.

An Iso Surface Extractor widget is used for performing real timeextraction of iso surfaces from image stacks. This may be helpful if anew image stack is imported, but no other software has been used tosegment a mesh from it yet. This widget will perform a simple marchingcubes iso value segmentation routine to extract the surface at aparticular value. The value it uses is set by pointing the focus arrowat the display image slice on the table. The value immediately under thetip of the arrow will be used as the iso value and a surfacerepresenting the boundary at that value will be extracted. The followingadditional controls are available: (a) Active Extract (check box)—Ifchecked, the iso surface will be continually extracted each time thewidget is moved. Due to the fact that the computation can be intensiveon very high resolution image stacks; (b) Invert (check box)—If afterextracting, the normals on the mesh are incorrect (i.e. the back facesshould be the front faces or the lighting looks incorrect), this checkbox can be used to switch the directions of the normals so the nextextraction is correct; (c) Select (button)—If pressed, the iso surfacewill extracted as described above. Only one iso surface can exist perframe, if an existing iso surface is in the frame, it will be replaced.

A Screen Recorder widget is used to capture video from the screens forreplaying externally after an interactive session. The file names areauto-generated depending on the timestamp, additionally, they will havea modifier in their name (left, right, or table) to indicate if they arethe left eye of the stereo screen, right eye, or a capture of the tablesurface. To capture interaction, the rendered frames are encoded into avideo. This encoding has an associated frame rate at which the videowill be played back. For the video to play back at the proper real timespeed, the frame rate of the encoding should ideally match the framerate of the render graphics. A close default value is used, butdepending on what is being recorded and other system load, this valuemay change. Thus, ideally the video frame rate should be set manually toan appropriate value. This frame rate control and other related settingsare available via the widget: (a) Average FPS—This field will report arunning average of the current rendering frame rate, this is ideally thevalue that should be entered into the encoding FPS field. However, thisvalue will decrease after recording is started, as the rendering timeadds significant additional load to the system causing slow down; (b)Record in Stereo (check box)—If checked, both the left and right eyeviews will be saved out to two separate video files. Third partysoftware (e.g. Bino) can then be used to compose these two video filesand play it back in stereo. If unchecked, only the left eye is saved;(b) Record Table (check box)—If checked, a video file will be createdthat captures the graphics rendered on the table surface; (c) EncodingFPS—A modifiable field that tells the video encoder what frame rate toencode at, ideally this should match the value reported in the AverageFPS field (note this value will decrease after recording is beganthough); (d) Start/Stop Recording (button)—When clicked, the videorecording will start. When unclicked, the recording stops and the videosare saved to disk.

The Mesh Loader widget is used to load new triangle meshes into thesoftware. These triangle meshes can be added to existing frames, or newframes can be created to contain their own separate meshes. Using theexpanded load window, a file can be browsed to and selected, uponselection the file will be read and the mesh will be displayed. Only asingle file can be loaded at a time, to load multiple meshes, load eachone sequentially. The main supported file format is STL, which most CADpackages support exporting to. Additionally, many other triangle meshformats work as well, for example Wavefront OBJ files.

The Temporal Mesh Loader widget is used to load sequences of trianglemeshes that are temporally consecutive. A sequence is loaded simply byspecifying the list of triangle mesh files to load (where the order ofthe filenames is temporally consistent). This widget works almostidentically to the “Mesh Loader”, however in the load window multiplefiles should be selected that define the sequence, instead of just one.

The Volume Loader widget is used to load image stacks (i.e. volumes)into frames. Exactly one image stack (or sequence of image stacks) canbe loaded per frame. If the stack is loaded into an existing frame itwill replace any existing image stack, otherwise a new frame will becreated. The load window can be expanded and a list of files can beselected that contain the image stack to load. Dicom images aresupported, but most other image formats are also supported, png files(that also support color) typically are more robust. The images will beloaded in the order they are listed in the file browser (which isalphabetically sorted). The arrow at the top of the file browser windowcan be used to reverse the ordering of the filenames. Additionally,there is also a “bookend” feature, where the starting file and endingfile can be selected, and after pressing this “bookend” button at thetop, all in between files will be selected. Finally, temporal sequencesof image stacks can be loaded with this widget as well. To do this, eachstack should contain the exact same number of images, and the imagesthat compose all of the sequential images stacks should be located in asingle folder, such that the first image in a subsequent stackimmediately follows the last image in the previous stack. Then using theadditional settings, the interval of how to break up this large list offiles can be specified. The specific additional settings are as follows:(a) Load Empty (button)—If pressed, a new frame with an empty volumewill be created; (b) Temporal Volume (check box)—If checked, the list offiles will be loaded as a temporal sequence, as described above; (c)Slice per Timestep—The interval that distinguishes sequential imagestacks from one another, for example if 200 images were loaded, andthere were 10 sequential volumes in it, 20 should be entered here.

The Project Loader widget is used to save out the entire state of thesystem as it currently is. This is helpful if multiple meshes andvolumes have been loaded, and different settings have been applied, andyou wish to save it for displaying at another later time. By expandingthe save window and selecting a directory, a .proj file will be savedcontaining all the information needed to reload the software in theexact same state as which it was saved. This .proj file is in plain textand can be examined in a text editor if you wish to change someindividual settings manually. To reload a saved project file, expand theload window, browse to it, and select it. It may take a few minutes toload all the files linked in this project. For the project load to besuccessful, the files that were used when the .proj was saved mustreside in the exact same location. Additionally, sometimes certainfilenames may break the parser that is used to load this file. If unableto reload a project file, it is most likely due to the fact that thefile names had some unusual characters (or spaces) in their paths thatare breaking the parsing. For this reason, simple filenames anddirectories are recommended.

The Orthogonal Slice Viewer widget is used to view three, orthogonal,coordinate aligned 2D slices of the current volume. When activated,these three views will be displayed on the left side of the tablesurface. The orientations of these three views are locked to the localframe of the virtual objects, i.e. as the objects are rotated relativeto the global world, the 2D slices remain the same. The current centerpoint of the view window determines the 3D location of each of thesethree slices. This single point will always intersect all three of theslices. As the user navigates via the view window, the orthogonal slicesupdate dynamically in response to the changing of this point. Whenactive, the exact 3D locations of these slices will be shown in theWorld-In-Miniature as three separate frames. The following settings areavailable for this widget: (a) Show Slices (check box)—When checked, theslices are active and will displayed; (b) Mirror on Wall (check box)—Ifchecked, and the slice viewer is active, the 2D slice views will bemirror on the vertical surface.

The Temporal FEA Loader widget is used to load in FEA data that followsthe PLT format. This format lists the elements and their connectivity,followed by a series of time steps that contains the position of thevertices of all the elements and a number of data fields associated withthese vertices. The file load window is used to browse to the locationof a PLT file and after selected a set of time varying meshes is loadedfrom the file. After being loaded, the visual display of these meshescan be modified by changing the following settings: (a) Local Color Map(check box)—When checked, the mesh is colored such that the range ofvalues is mapped to the maximum and minimum field values for the currenttime step. If off, a global color map is used where the range is definedby the field is determined across all time steps; (b) Flat Shading(check box)—If checked, when a PLT file is loaded the normals arecomputed using a flat shading local to each triangle instead of a smoothshading, useful if the mesh contains sharp edges; (c) Mesh Lighting On(check box)—When unchecked, the lighting is disabled for the currentdisplayed FEA meshes; (d) Wireframe On (check box)—When checked, turnson a wireframe rendering for the current displayed FEA meshes; (e)Displayed Field (drop down)—Contains a list of all the fields that wereloaded from the PLT file. After selected the displayed field is updatedto reflect this field; (f) Color Map (drop down)—Contains a list ofpossible color mappings to use when rendering the displayed field. Afterselected the color of the field on the meshes is updated; (g) SkipInterval—It is possible that the set of all FEA meshes contained in thePLT file is too large to load into memory for the PC all at once. To beable to still load a representation of the entire dataset the skipinterval can set and timesteps will be skipped to reduce the amount ofdata loaded.

The following section describes selected aspects of the program,according to one example.

Low-Level Strategies for Widgets/Controlling the 3D Touch System:

One example allows for loading and simultaneously displaying multipledatasets. To realize the feature of supporting loading andsimultaneously displaying multiple datasets (e.g., 3D surface,volumetric FEA data) notice that many widgets only affect one dataset ata time. This means that whenever a widget is activated it needs to beassociated with the particular dataset upon which it should act. Thefunction of loading and simultaneously displaying multiple datasetsincludes a new interface. To make the association between widget anddataset, the user drags the widget from the tool tray onto the shadow ofthe appropriate dataset. Then the widget activates at that position,right on top of the dataset's shadow. In one example, widgets that applyto all datasets (e.g., loading and saving an entire project, recordingthe screen to a movie file) are activated by dragging and dropping themanywhere on the table surface (and not necessarily on top of a dataset).In one example, widgets for loading a new dataset are activated bydragging them to a new position on the table, releasing, and then thenew dataset will be loaded up at this location on the table. Similarly,in the camera path tool, the camera is associated with a particular 3Dcurve to fly along by dragging the camera widget onto the shadow of the3D curve.

Centering a 3D Curve Relative to the Anatomy;

In one example, a spline can be centered relative to selected anatomy.To support the feature of centering a spline (3D curve) that has beenplotted inside the cavity of a 3D anatomy, an algorithm iterates througheach control point of the spline, determines the distance from thispoint to the closest surface in all directions perpendicular to thespline, and then centers the control point based on these computeddistances. In one example, an algorithm is configured to compute theaverage distance from the spline to the anatomy along the length of thespline.

Real-Time Interactive Iso-Surface Extraction:

A significant amount of time is required in bringing new datasets intothe tool. Some of this time is associated with using an external tool toextract an iso-surface from medical image stacks. One example of thepresent subject matter includes an iso-surface extractor widget whicheliminates time spent on accessing an external tool. The widget isconfigured to perform a simple marching cubes iso value segmentation toextract the surface at a particular value. The value it uses is set bypointing the focus arrow at the display image slice on the table. Thevalue immediately under the tip of the arrow is used as the iso valueand a surface representing the boundary at that value is extracted. Inthe “Active Extract” mode, the iso surface is continually extracted eachtime the widget is moved, making this a real-time interface for dataexploration.

Generating a 3D Curve that Follows the Contour of Anatomy:

One example includes a tool to map a cavity and contour of a 3D surfacein an anatomical dataset. The tool executes an algorithm that uses anexisting user-created spline to select an intersection point with thevertical surface. Three dimension rays are originated at thisintersection point and radially projected outward to find theintersection with the anatomy. A new spline is created with theseintersection points as the set of control points.

Three-Axis-Aligned Orthogonal 2D Viewplanes of Volume Data:

One example includes a tool to support the visualization of thetraditional three axial aligned (sagittal, coronal, transverse) views.The tool is configured to display three slices on the left hand side ofthe table surface. The 3D location of each of these slices is determinedby a single point that intersects all three. This point is set by theuser as they navigate using the Slice WIM metaphor and is determined bythe 3D location of the bottom center of the view window (as indicated onthe tabletop widget).

Anatomical Fly-Through with Device Radius Sizing:

One example is configured to support camera fly-throughs relative to aset device size. This tool allows attachment of a camera widget to auser-created spline. After the camera widget is attached, the user cancontrol the playback of the fly-though by pausing, reversing direction,and speeding up or slowing down the animation. The view is set duringthe fly-through by moving the WIM “through” according to the motion ofthe spline, in this way the view window remains fixated and the geometrymoves through it, giving the effect of view motion. In addition, a userdefined radius can be set that shows the size of a device as circle onthe screen. The user can interpret the geometry relative to this devicesize to determine intersections and fit.

FIGS. 4M and 4N show a representation of a virtual desktop formanipulating and visualizing items such as anatomical items. In general,the representation shows features by which a user can manipulate avisual representation on an elevated desktop while other items such asicons are projected down into a virtual sub-desktop, an then have theitems from the sub-desktop visually transitioned so that they can beinteracted with more conveniently.

As shown in FIG. 4M, a human user 402 of a computer visualization systemis seated at the system and is interacting with an elevated desktop 406using a stylus 404. The elevated desktop 406 is implemented as acomputer display, such as an LCD display that may be connected tohardware like that described above and below for the modeling andvisualization of anatomical items. A “virtual table” 408 is drawn with3D graphics so that it appears to lie in the plane of a real tabletop410 lying beneath the elevated desktop that is sitting at an angle withrespect to the horizontal real tabletop 410. When the virtual table 408is in this horizontal position, it provides a visual grounding contextfor the 3D visualization on the virtual table 408. It also provides a 3Dsurface upon which the user 402 can view useful interaction widgets(buttons, color pickers, etc.). Such items may be visually behind theitem being manipulated (e.g., a 3D model of a human heart that appearsto user 402 as if it is floating in space at or about the plane of theelevated desktop 406. Therefore, the widgets can be presented withoutoccluding any of the data visualization

As shown in FIG. 4N, the virtual table has been automatically rotatedupward through visual manipulation of the 3D presentation so that it nowappears to be in the plane of the elevated desktop rather than in theplane of the real tabletop 410. Such manipulation may be preferred whenthe widgets visually located on the real tabletop 410 become difficultto select for the user 402 because they appear several inches away fromthe physical display surface. Since the user 402 cannot reach out anddirectly touch these widgets, a “virtual laser pointer” technique isusually used to select such out-of-reach objects, but control of such apointer can be challenging for some users. By allowing the user 402 toselect a user-selectable displayed item, and in response, causing thedisplay of the widgets and other menu items to visually rotate up fromthe lower displayed location to the higher location that is moredirectly correlated to the location the user 402 is touching, theinterface may be easier for the user 402 to employ.

In one implementation of such a desktop system in use, a method maycomprise: displaying on a slanted physical device positioned in anon-vertical and non-horizontal orientation, a three-dimensional modelthat appears to a user to be situated at or in front of the physicaldevice; displaying on a first virtual surface visually behind thephysical device a plurality of user-selected items that include icons,menus, or both; receiving a user input to change a displayed location ofthe user-selectable items; and in response to receiving the user input,transitioning the user-selectable items from the first virtual surfaceto be displayed as appearing to be on a surface of the slanted physicaldevice. The first virtual surface may be a horizontal surface, such as aphysical worktop on which the slanted physical device is resting. Thefirst virtual surface may also be displayed behind the three-dimensionalmodel so that when the three-dimensional model visually overlaps thefirst virtual surface, the three-dimensional model at least partiallyblocks display of the first virtual surface. The first virtual surfacemay also be displayed as a plane or as a curved surface. The displayeditems may be user-selectable in a manner that changes a display of thethree-dimensional model. The model may take the form ofthree-dimensional and four-dimensional models described above and below,and the user-interaction and processes described above and below may beemployed with a desktop system like that described here.

FIGS. 5A and 5B show schematically a computation of distance from edgesof an anatomical item to a 3D spline curve. Such steps reflected in thefigures may be carried out automatically by a visualization system suchas those discussed above and below so as to automatically place a splinein a most convenient physical location, such as to represent the centerpoint of a path that a medical device, such as a stent, may be guidedalong through the anatomical item. The techniques described here can beused to determine the distance of an existing spline from the interiorwalls of an anatomical item like vasculature, or to locate a spline thatis to be at a defined position relative to the internal wall, such as aspline that is to be centered on a length of vasculature.

FIG. 5A shows a point 506A on a spline at which measurements are to bemade for determining distance to a wall 502 of a vessel or internal wallof a human heart. A computational method in this example identifies thepoint 506A as being a point along a spline curve, and generates aplurality of arrays around the spline at an orthogonal angle to thespline at that point of the alpine. (Note that a second wall of theanatomical item is not shown, but would be located to the left of thespline.) The arrays may be generated at every n degrees around thespline, where n typically can be divided evenly into 360, such as every10, 30, 60, or 120 degrees. The spacing between the rays may also beuneven. The distance from point 506A to the wall of the anatomical itemmay be computed for each such array and stored. As shown by FIG. 5B, themeasurement process may be repeated for a next point along a length ofthe alpine, point 506B. In each instance, the arrays may draw out aninternal outline of the shape of the anatomical item at the point, as508A and 508B. The process may then be repeated for each of theidentified points along the spline, where the points may be nequally-spaced or unequally-spaced points along the spline.

In this manner, various computations and determinations may be made. Forexample, the cross sectional shape and area of the item can bedetermined, where the cross-section is roughly perpendicular to thepassage at that point, rather than a horizontal slice through the itemthat may not adequately represent the geometry of the item. Suchgeometry may also be used to determine interaction between a medicaldevice, which may subsequently located at each relevant point along thespline so as to model movement of the device through the anatomicalitem, and the tissue around the medical device at the relevantlocations.

A spline may be automatically fit to a passage in a similar manner. Asjust one example, a user may identify a passage through which he wouldlike a spline to be generated, such as a spline that closely follows themiddle of the passage. A system may then step through a model of theitem layer-by-layer, and compute a center of the opening (which forvasculature may be roughly a circle) at that layer, via standardtechniques. Such a step may result in the creation of a plurality ofpoints along a length of the passage, and a preliminary spline may befit to those points. However, such a spline may not best track acenterline of the item if it curves along its length. As a result, thetechniques just described may be applied to the preliminary spline, withthe points that define the spline being adjusted in position until therays are approximately equal in length around the spline. A final splinemay then be generated from those points, which, rather than being simplyequidistant from the walls of the item in a horizontal plane, areequidistant from the spline in a plane that is perpendicular to the axisof the spline at each relevant point along the spline.

FIG. 5 shows an example of a computer system that can be used toimplement the techniques described here. The system 500 can be used forthe operations described in association with any of thecomputer-implement methods described previously, according to oneimplementation. The system 500 is intended to include various forms ofdigital computers, such as laptops, desktops, workstations, personaldigital assistants, servers, blade servers, mainframes, and otherappropriate computers. The system 500 can also include mobile devices,such as personal digital assistants, cellular telephones, smartphones,and other similar computing devices. Additionally the system can includeportable storage media, such as, Universal Serial Bus (USB) flashdrives. For example, the USB flash drives may store operating systemsand other applications. The USB flash drives can include input/outputcomponents, such as a wireless transmitter or USB connector that may beinserted into a USB port of another computing device.

The system 500 includes a processor 510, a memory 520, a storage device530, and an input/output device 540. Each of the components 510, 520,530, and 540 are interconnected using a system bus 550. The processor510 is capable of processing instructions for execution within thesystem 500. The processor may be designed using any of a number ofarchitectures. For example, the processor 510 may be a CISC (ComplexInstruction Set Computers) processor, a RISC (Reduced Instruction SetComputer) processor, or a MISC (Minimal Instruction Set Computer)processor.

In one implementation, the processor 510 is a single-threaded processor.In another implementation, the processor 510 is a multi-threadedprocessor. The processor 510 is capable of processing instructionsstored in the memory 520 or on the storage device 530 to displaygraphical information for a user interface on the input/output device540.

The memory 520 stores information within the system 500. In oneimplementation, the memory 520 is a computer-readable medium. In oneimplementation, the memory 520 is a volatile memory unit. In anotherimplementation, the memory 520 is a non-volatile memory unit.

The storage device 530 is capable of providing mass storage for thesystem 500. In one implementation, the storage device 530 is acomputer-readable medium. In various different implementations, thestorage device 530 may be a floppy disk device, a hard disk device, anoptical disk device, or a tape device.

The input/output device 540 provides input/output operations for thesystem 500. In one implementation, the input/output device 540 includesa keyboard and/or pointing device. In another implementation, theinput/output device 640 includes a display unit for displaying graphicaluser interfaces.

The features described can be implemented in digital electroniccircuitry, or in computer hardware, firmware, software, or incombinations of them. The apparatus can be implemented in a computerprogram product tangibly embodied in an information carrier, e.g., in amachine-readable storage device for execution by a programmableprocessor; and method steps can be performed by a programmable processorexecuting a program of instructions to perform functions of thedescribed implementations by operating on input data and generatingoutput. The described features can be implemented advantageously in oneor more computer programs that are executable on a programmable systemincluding at least one programmable processor coupled to receive dataand instructions from, and to transmit data and instructions to, a datastorage system, at least one input device, and at least one outputdevice. A computer program is a set of instructions that can be used,directly or indirectly, in a computer to perform a certain activity orbring about a certain result. A computer program can be written in anyform of programming language, including compiled or interpretedlanguages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment.

Suitable processors for the execution of a program of instructionsinclude, by way of example, both general and special purposemicroprocessors, and the sole processor or one of multiple processors ofany kind of computer. Generally, a processor will receive instructionsand data from a read-only memory or a random access memory or both. Theessential elements of a computer are a processor for executinginstructions and one or more memories for storing instructions and data.Generally, a computer will also include, or be operatively coupled tocommunicate with, one or more mass storage devices for storing datafiles; such devices include magnetic disks, such as internal hard disksand removable disks; magneto-optical disks; and optical disks. Storagedevices suitable for tangibly embodying computer program instructionsand data include all forms of non-volatile memory, including by way ofexample semiconductor memory devices, such as EPROM, EEPROM, and flashmemory devices; magnetic disks such as internal hard disks and removabledisks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Theprocessor and the memory can be supplemented by, or incorporated in,ASICs (application-specific integrated circuits).

To provide for interaction with a user, the features can be implementedon a computer having a display device such as a CRT (cathode ray tube)or LCD (liquid crystal display) monitor for displaying information tothe user and a keyboard and a pointing device such as a mouse or atrackball by which the user can provide input to the computer.Additionally, such activities can be implemented via touchscreenflat-panel displays and other appropriate mechanisms.

The features can be implemented in a computer system that includes aback-end component, such as a data server, or that includes a middlewarecomponent, such as an application server or an Internet server, or thatincludes a front-end component, such as a client computer having agraphical user interface or an Internet browser, or any combination ofthem. The components of the system can be connected by any form ormedium of digital data communication such as a communication network.Examples of communication networks include a local area network (“LAN”),a wide area network (“WAN”), peer-to-peer networks (having ad-hoc orstatic members), grid computing infrastructures, and the Internet.

The computer system can include clients and servers. A client and serverare generally remote from each other and typically interact through anetwork, such as the described one. The relationship of client andserver arises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular implementations of particularinventions. Certain features that are described in this specification inthe context of separate implementations can also be implemented incombination in a single implementation. Conversely, various featuresthat are described in the context of a single implementation can also beimplemented in multiple implementations separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

Other features may be implemented using the features discussed above,though they may not be part of the initially-filed claims. For example,A computer-implemented medical visualization method is disclosed thatcomprises identifying a three-dimensional model of an anatomical item ofa particular mammal; identifying a force applied to tissue of theanatomical items; determining stress and strain on the tissue of theanatomical item using finite element analysis of the three-dimensionalmodel; and displaying the anatomical item and a visual representation ofstrains determined for the tissue of the anatomical item as a result ofthe identified force. The anatomical item can comprise portions of aheart and the force applied on the anatomical item is identified basedon movement of a virtual medical device through the heart. The virtualmedical device can comprise a three-dimensional model that is modeled tohave flexibility in at least one dimension. The method can also includeautomatically determining whether the virtual medical device makescontact with tissue in the three-dimensional model of the anatomicalitem as the virtual medical device is moved relative to thethree-dimensional model of the heart. The method can additionally oralternatively include performing computational fluid dynamics analysisfor fluid flowing around the medical device in the three-dimensionalmodel of the heart.

In some aspects, the three-dimensional model of the anatomical item isgenerated by operations comprising: obtaining data that representsimaged two-dimensional slices of the anatomical item captured by one ormore medical imaging systems; generating a three-dimensional mesh frompoints identified in the two-dimensional slices; and generating thethree-dimensional model from the three-dimensional grid. The initialmethod may also comprise displaying an animation of thethree-dimensional model of the anatomical item, the animation comprisedof frames taken by imaging a real version of the anatomical item atdifferent closely-related times, so as to show movement of theanatomical item. The method also may include showing differences in theanatomical item between particular ones of the frames computed by finiteelement analysis, and alternatively or additionally receiving an inputfrom the user identifying locations in the three-dimensional model ofthe anatomical item, and displaying to the user distances in the modelrepresented by the identified locations. Displaying the visualrepresentation of strains determined for the tissue of the anatomicalitem as a result of the identified force can comprise displaying theanatomical item in different colors that each represent a particularlevel of strain.

In yet another implementations, a computer-implemented medicalvisualization method may be implemented and comprise: identifying athree-dimensional model of an anatomical item of a particular mammal;displaying a moving animation of the three-dimensional model, the movinganimation created from multiple frames from imaging of the anatomicalitem over a sort time period to capture movement of the anatomical item;displaying one or more non-moving views of the three-dimensional modelwhile the moving animation is being displayed; and in response to inputsfrom a user, changing the displayed moving animation and non or morenon-moving views in coordination with each other. The method can alsoinclude displaying, in coordination between the moving animation and theone or more non-moving views, an device being moved through theanatomical item. The device can comprise a medical device, representedby a three-dimensional model, being moved through open passages in theanatomical item, and the anatomical item can comprise a heart. Themethod may additionally comprise displaying forces applied on theanatomical item based on movement of the device through the anatomicalitem. The displayed forces can be determined by applying finite elementanalysis to a tissue model for the anatomical item. Also, the initialprocess may comprise automatically determining whether the medicaldevice makes contact with tissue in anatomical item as the medicaldevice is moved relative to the three-dimensional model of theanatomical item. The method can additionally include performingcomputational fluid dynamics analysis for fluid flowing around themedical device in the anatomical item, and displaying to the userinformation representing results of the computational fluid dynamicsanalysis.

In certain aspects, the three-dimensional model of the anatomical itemis generated by operations comprising: obtaining data that representsimaged two-dimensional slices of the anatomical item captured by one ormore medical imaging systems; generating a three-dimensional mesh frompoints identified in the two-dimensional slices; and generating thethree-dimensional model from the three-dimensional grid. Also, themoving animation can be comprised of frames taken by imaging a realversion of the anatomical item at different closely-related times, so asto show movement of the anatomical item. In addition, a first non-movingview shows the anatomical item during a first imaging session, and asecond non-moving view shows the anatomical item during a second imagingsession. Moreover, the first imaging session can be a pre-operativeimaging session, and the second imaging session can be a post-operativeimaging session.

In another implementation, a computer-implemented medical visualizationmethod is disclosed that comprises identifying first and secondthree-dimensional models of an anatomical item of a particular mammal,wherein the first model is formed from imaging data taken at a firstimaging session, and the second model is formed from imaging data takenat a separate, second imaging session; displaying the first model on acomputer-generated visual display; and displaying the second model onthe computer-generated visual display concurrently with displaying thefirst model. The method can also comprise receiving a user input tomanipulate one of the first or second model, or both, and changing aview of the first model and a view of the second model in coordinationwith each other responsive to the received user input. The firstthree-dimensional model can represent an anatomical item pre-operation,and the second three-dimensional model can represent the anatomical itempost-operation. The anatomical item can comprise a heart or other organ.

In some aspects, the method further comprises displaying the first andsecond three-dimensional models as animations showing motion of theanatomical item, wherein the animations are displayed in time-wisecoordination with each other. The animations can be displayed asrepeating, looping animations, and views of the models that aredisplayed change in response to user input. The method can also includedisplaying forces applied on the anatomical item based on movement of avirtual device through the one or more of the models of the anatomicalitem. In some aspects, the three-dimensional model of the anatomicalitem is generated by operations comprising: obtaining data thatrepresents imaged two-dimensional slices of the anatomical item capturedby one or more medical imaging systems; generating a three-dimensionalmesh from points identified in the two-dimensional slices; andgenerating the three-dimensional model from the three-dimensional grid.

In yet another implementation, a computer-implemented medicalvisualization method is disclosed that comprises identifying athree-dimensional model of an anatomical item of a particular mammal;displaying a view of the three-dimensional model on a computer-basedvisualization system; receiving input from a user identifying at leasttwo points on the displayed view of the three-dimensional model; andproviding to the user data that indicates a measurement based on theidentified at least two points, the measurement representing a scale onthe anatomical item. The method can also comprise selecting a devicesized according to the measurement, and displaying the device in theanatomical item. Displaying the device in the anatomical item cancomprise moving the device along a path defined in open passages of thethree-dimensional model. Also, the device can comprise a medical device,represented by a three-dimensional model, being moved through openpassages in the anatomical item, and the anatomical item can comprise aheart or other animal (e.g., human) organ. The method can also includedisplaying forces applied on the anatomical item based on movement ofthe device through the anatomical item, and the displayed forces can bedetermined by applying finite element analysis to a tissue model for theanatomical item. Moreover, the method can include displaying a movinganimation of the three-dimensional model and presenting to the userchanging measurements as the animation moves.

In some aspects, the measurement represents a radius, diameter, area, orvolume of an open space in the anatomical item. Also, thethree-dimensional model of the anatomical item can be generated byoperations comprising: obtaining data that represents imagedtwo-dimensional slices of the anatomical item captured by one or moremedical imaging systems; generating a three-dimensional mesh from pointsidentified in the two-dimensional slices; and generating thethree-dimensional model from the three-dimensional grid.

In another implementation, a computer-implemented medical visualizationmethod is disclosed that comprises identifying first data representingan anatomical item of a particular mammal, the first data obtained by afirst type of medical imaging; identifying second data representing theanatomical item, the second data obtained by a second type of medicalimaging that is different from the first type of medical imaging; andbuilding a three-dimensional solid model of the anatomical item using acombination of the first data and the second data. The first data canrepresent slices of the anatomical item at multiple differentelevations, and the second data can represent slides of the anatomicalitem at multiple different elevations. Moreover, the first data and thesecond data can represent slices that are determined to be essentiallyparallel to each other. The first type of medical imaging can comprisemagnetic resonance imaging, and the second type of medical imaging cancomprise computer tomography imaging. The method may also includedisplaying, in coordination, a moving animation of the anatomical itemgenerated from the three-dimensional solid model and a non-moving viewof the anatomical item generated from the three-dimensional solid model,and change views of the animation and the non-moving view based on userinputs received by a computer visualization system. The anatomical itemcan comprise a heart or other organ. The method can also comprisedisplaying forces applied on the anatomical item based on movement of adevice through the three-dimensional model. Also, the displayed forcescan be determined by applying finite element analysis to a tissue modelfor the anatomical item.

In some aspects, the three-dimensional model of the anatomical item isgenerated by operations comprising: obtaining data that representsimaged two-dimensional slices of the anatomical item captured by thefirst and second types of medical imaging; generating athree-dimensional mesh from points identified in the two-dimensionalslices; and generating the three-dimensional model from thethree-dimensional grid. The method can also include building a secondthree-dimensional model of the anatomical item using data captured inseparate sessions of the first type of medical imaging and the secondtype of medical imaging, wherein the separate sessions are differentthan sessions used to generate data for the initial three-dimensionalmodel.

In another implementation, a computer-implemented medical visualizationmethod, is disclosed that comprises identifying a three-dimensionalmodel of an anatomical item of a particular mammal; displaying thethree-dimensional model on a computer-based visualization system;receiving input from a user to add items in positional relationship withthe three-dimensional model; and storing meta data defining the addeditems in association with the three-dimensional model so that the addeditems can be automatically displayed with the three-dimensional modelwhen the three-dimensional model is accessed in the future. Thethree-dimensional model of the anatomical item is generated byoperations can comprise: obtaining data that represents imagedtwo-dimensional slices of the anatomical item captured by one or moremedical imaging systems; generating a three-dimensional mesh from pointsidentified in the two-dimensional slices; and generating thethree-dimensional model from the three-dimensional grid.

In another implementation, a computer-implemented medical visualizationmethod comprises identifying a three-dimensional model of an anatomicalitem of a particular mammal; displaying the three-dimensional model on acomputer-based visualization system; receiving input from a userdefining multiple non-parallel cutting planes; and displaying thethree-dimensional model with portions of the three-dimensional modelremoved as defined by the multiple non-parallel cutting planes. Themultiple non-parallel cutting planes can define a corner of a box thatis overlaid with a portion of the three-dimensional model so as to cut acorner visually into the three-dimensional model. The anatomical itemcan comprise a heart, and the three-dimensional model of the anatomicalitem can be generated by operations comprising: obtaining data thatrepresents imaged two-dimensional slices of the anatomical item capturedby one or more medical imaging systems; generating a three-dimensionalmesh from points identified in the two-dimensional slices; andgenerating the three-dimensional model from the three-dimensional grid.

In yet another implementation, a computer-implemented medicalvisualization method is disclosed that comprises identifying athree-dimensional model of an anatomical item of a particular mammal;identifying different sub-portions of the three-dimensional model basedon physiological roles placed by the different sub-portions; displayingthe three-dimensional model on a computer-based visualization system;and presenting user-selectable controls for manipulating thethree-dimensional model on the basis of the identified differentsub-portions. The different sub-portions are assigned to defined layersof the three-dimensional model that can be separately activated, coloredwith a defined color, or both. The anatomical item can comprise a heart,and the three-dimensional model of the anatomical item can be generatedby operations comprising: obtaining data that represents imagedtwo-dimensional slices of the anatomical item captured by one or moremedical imaging systems; generating a three-dimensional mesh from pointsidentified in the two-dimensional slices; and generating thethree-dimensional model from the three-dimensional grid.

In another implementation, a computer-implemented medical visualizationmethod is disclosed that comprises identifying a three-dimensional modelof an anatomical item of a particular mammal; displaying thethree-dimensional model on a computer-based visualization system of auser who is viewing the actual anatomical item; and changing the displayof the three-dimensional model automatically in response to interactionby the user with the actual anatomical item. The computer-basedvisualization system can comprises electronic glasses worn by the user,and the interaction by the user can comprise inserting a medical deviceinto the particular mammal, and changing the display comprises showing arepresentation of progress of the device advancing internal to theanatomical item. The anatomical item can comprise a portion of acirculatory system, and progress of the device can comprise progress ofadvancing an implantable cardiac device at a location in the circulatorysystem. The anatomical item can comprise a heart or other organ, or aportion of vasculature.

Thus, particular implementations of the subject matter have beendescribed. Other implementations are within the scope of the followingclaims. In some cases, the actions recited in the claims can beperformed in a different order and still achieve desirable results. Inaddition, the processes depicted in the accompanying figures do notnecessarily require the particular order shown, or sequential order, toachieve desirable results. In certain implementations, multitasking andparallel processing may be advantageous.

What is claimed is:
 1. A computer-implemented medical visualization method, comprising: identifying a three-dimensional model of an anatomical item of a particular mammal; displaying a moving animation of the three-dimensional model, the moving animation created from multiple frames from imaging of the anatomical item over a short time period to capture movement of the anatomical item; displaying one or more non-moving views of the three-dimensional model while the moving animation is being displayed; and in response to receiving inputs from a user, changing the displayed moving animation and the one or more non-moving views automatically in coordination with each other.
 2. The method of claim 1, further comprising displaying, in coordination between the moving animation and the one or more non-moving views, a medical device being moved through the anatomical item.
 3. The method of claim 2, wherein the anatomical item comprises a heart.
 4. The method of claim 2, further comprising displaying forces applied on the anatomical item based on movement of the device through the anatomical item.
 5. The method of claim 4, wherein the displayed forces are determined by applying finite element analysis to a tissue model for the anatomical item.
 6. The method of claim 2, further comprising automatically determining whether the medical device makes contact with tissue in the anatomical item as the medical device is moved relative to the three-dimensional model of the anatomical item.
 7. The method of claim 1, further comprising performing computational fluid dynamics analysis for fluid flowing around the medical device in the anatomical item, and displaying to the user information representing results of the computational fluid dynamics analysis.
 8. The method of claim 1, wherein the three-dimensional model of the anatomical item is generated by operations comprising: obtaining data that represents imaged two-dimensional slices of the anatomical item captured by one or more medical imaging systems; generating a three-dimensional intermediate representation from points identified in the two-dimensional slices; and generating the three-dimensional model from the intermediate three-dimensional representation.
 9. The method of claim 1, wherein the moving animation is comprised of frames taken by imaging a real version of the anatomical item at different closely-related times, so as to show movement of the anatomical item.
 10. The method of claim 1, wherein a first non-moving view shows the anatomical item during a first imaging session, and a second non-moving view shows the anatomical item during a second imaging session.
 11. The method of claim 10, wherein the first imaging session is a pre-operative imaging session, and the second imaging session is a post-operative imaging session.
 12. A computer-implemented medical visualization method, comprising: identifying first and second three-dimensional models of an anatomical item of a particular mammal, wherein the first model is formed from imaging data taken at a first imaging session, and the second model is formed from imaging data taken at a separate, second imaging session; displaying the first model on a computer-generated visual display; and displaying the second model on the computer-generated visual display concurrently with displaying the first model.
 13. The method of claim 12, further comprising receiving a user input to manipulate one of the first or second model, or both, and changing a view of the first model and a view of the second model in coordination with each other responsive to the received user input.
 14. The method of claim 12, wherein the first three-dimensional model represents an anatomical item pre-operation, and the second three-dimensional model represents the anatomical item post-operation.
 15. The method of claim 12, wherein the anatomical item comprises a heart.
 16. The method of claim 12, further comprising displaying the first and second three-dimensional models as animations showing motion of the anatomical item, wherein the animations are displayed in time-wise coordination with each other.
 17. The method of claim 16, wherein the animations are displayed as repeating, looping animations, and views of the models that are displayed change in response to user input.
 18. The method of claim 12, further comprising displaying forces applied on the anatomical item based on movement of a virtual device through the one or more of the models of the anatomical item.
 19. The method of claim 12, wherein the three-dimensional model of the anatomical item is generated by operations comprising: obtaining data that represents imaged two-dimensional slices of the anatomical item captured by one or more medical imaging systems; generating a three-dimensional intermediate representation from points identified in the two-dimensional slices; and generating the three-dimensional model from the three-dimensional intermediate representation.
 20. A computer-implemented medical visualization method, comprising: identifying a three-dimensional model of an anatomical item of a particular mammal; receiving, at a first geographic site, first user input directed to the three-dimensional model and reflecting, via visual presentation of the three-dimensional model, changes to the three-dimensional model at the first geographic site and a second geographic site that is remote from the first geographic site; receiving, at the second geographic site, second user input directed to the three-dimensional model, and reflecting changes to the three-dimensional model at the second geographic site and the first geographic site.
 21. The computer-implemented medical visualization method of claim 20, further comprising displaying a three-dimensional version of the model adjacent an animated four-dimensional version of the model at both the first and second sites, and affecting a display of one of the three-dimensional version of the model and the four-dimensional version of the model in response to user interaction with only one of the three-dimensional version of the model and the four-dimensional version of the model. 